Issue with distributed when bokeh is installed #2376
Description
I seem to be having significant bugs with dask.distributed after I install bokeh.
I just installed a fresh miniconda3. After running pip install dask distributed, I can run:
$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
(ins)>>> from dask.distributed import Client
(ins)>>> client = Client()
(ins)>>> client
<Client: scheduler='tcp://127.0.0.1:60321' processes=32 cores=32>
(ins)>>> def inc(x):
(ins)... return x + 1
(ins)...
(ins)>>> def add(x, y):
(ins)... return x + y
(ins)...
(ins)>>> a = client.submit(inc, 10) # calls inc(10) in background thread or process
(ins)>>> b = client.submit(inc, 20) # calls inc(20) in background thread or process
(ins)>>> a
<Future: status: finished, type: int, key: inc-329d1c0d6d5adf260702a5cc673b66c5>
(ins)>>> a.result()
11
(ins)>>> exit()
tornado.application - ERROR - Exception in Future <Future cancelled> after timeout
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 970, in error_callback
future.result()
concurrent.futures._base.CancelledError
(Not sure what that last error is about.)
After this I do pip install bokeh and try again:
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
(ins)>>> from dask.distributed import Client
(ins)>>> client = Client()
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
distributed.nanny - WARNING - Worker process 15692 exited with status 1
distributed.nanny - WARNING - Worker process 15670 exited with status 1
distributed.nanny - WARNING - Restarting worker
distributed.nanny - WARNING - Restarting worker
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
...
OpenBLAS blas_thread_init: pthread_create: Resource temporarily unavailable
OpenBLAS blas_thread_init: RLIMIT_NPROC 1024 current, 2065958 max
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 250, in main
pid = os.fork()
BlockingIOError: [Errno 11] Resource temporarily unavailable
distributed.nanny - WARNING - Worker process 15662 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15648 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15652 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15680 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15656 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15688 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15696 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15700 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15664 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15646 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15674 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15668 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15642 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15682 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15686 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15694 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15698 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15702 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15678 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15672 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15660 was killed by unknown signal
distributed.nanny - WARNING - Worker process 15684 was killed by unknown signal
...
distributed.nanny - ERROR - Failed to restart worker after its process exited
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/nanny.py", line 291, in _on_exit
yield self.instantiate()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/nanny.py", line 226, in instantiate
self.process.start()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1141, in run
yielded = self.gen.throw(*exc_info)
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/nanny.py", line 370, in start
yield self.process.start()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1133, in run
value = future.result()
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/process.py", line 35, in _call_and_set_future
res = func(*args, **kwargs)
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/process.py", line 184, in _start
process.start()
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/process.py", line 112, in start
self._popen = self._Popen(self)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/context.py", line 291, in _Popen
return Popen(process_obj)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/popen_forkserver.py", line 35, in __init__
super().__init__(process_obj)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/popen_fork.py", line 20, in __init__
self._launch(process_obj)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/popen_forkserver.py", line 55, in _launch
self.pid = forkserver.read_signed(self.sentinel)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 312, in read_signed
raise EOFError('unexpected EOF')
EOFError: unexpected EOF
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
self = reduction.pickle.load(from_parent)
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 261, in main
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
old_handlers)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/forkserver.py", line 297, in _serve_one
code = spawn._main(child_r)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/spawn.py", line 115, in _main
self = reduction.pickle.load(from_parent)
File "/home/ipetrik/miniconda3/lib/python3.7/multiprocessing/synchronize.py", line 111, in __setstate__
self._semlock = _multiprocessing.SemLock._rebuild(*state)
FileNotFoundError: [Errno 2] No such file or directory
...
tornado.application - ERROR - Multiple exceptions in yield list
Traceback (most recent call last):
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 883, in callback
result_list.append(f.result())
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/tornado/gen.py", line 1147, in run
yielded = self.gen.send(value)
File "/home/ipetrik/miniconda3/lib/python3.7/site-packages/distributed/deploy/local.py", line 217, in _start_worker
raise gen.TimeoutError("Worker failed to start")
tornado.util.TimeoutError: Worker failed to start
if I try creating a dask-scheduler manually and connecting to it works just fine with bokeh.
Running with processes=False works, but obviously, I need processes in production.
$ python
Python 3.7.0 (default, Jun 28 2018, 13:15:42)
[GCC 7.2.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
(ins)>>> from dask.distributed import Client
(ins)>>> client = Client(processes=False)
(ins)>>> def inc(x):
(ins)... return x + 1
(ins)...
(ins)>>> def add(x, y):
(ins)... return x + y
(ins)...
(ins)>>> a = client.submit(inc, 10) # calls inc(10) in background thread or process
(ins)>>> a
<Future: status: finished, type: int, key: inc-329d1c0d6d5adf260702a5cc673b66c5>
(ins)>>> a.result()
11
(ins)>>> exit()
The environment is the head node of an HPC running CentOS Linux version 2.6.32-358.11.1.el6.x86_64 (mockbuild@c6b7.bsys.dev.centos.org). I do not have root access.