-
-
Notifications
You must be signed in to change notification settings - Fork 757
Closed
Labels
bugSomething is brokenSomething is broken
Description
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Task exception was never retrieved
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: future: <Task finished name='Task-103' coro=<Server._handle_comm() done, defined at /opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py:675> exception=TypeError("argument of type 'NoneType' is not iterable")>
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Traceback (most recent call last):
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: During handling of the above exception, another exception occurred:
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: Traceback (most recent call last):
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 769, in _handle_comm
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: result = await result
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 1692, in get_data
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: response = await comm.read(deserializers=serializers)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: convert_stream_closed_error(self, e)
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: if "UNKNOWN_CA" in exc.reason:
Aug 11 13:37:25 ip-10-0-4-2 cloud-init[1264]: TypeError: argument of type 'NoneType' is not iterable
(https://cloud.coiled.io/dask-engineering/clusters/48659/details on hendrik-debug-worker-e73a905446)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: 2022-08-11 14:00:05,617 - distributed.core - ERROR - argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 426, in read_bytes
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: self._try_inline_read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 841, in _try_inline_read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: self._check_closed()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 1017, in _check_closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: raise StreamClosedError(real_error=self.error)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: During handling of the above exception, another exception occurred:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 821, in handle_stream
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: msgs = await comm.read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: convert_stream_closed_error(self, e)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: if "UNKNOWN_CA" in exc.reason:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: TypeError: argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: 2022-08-11 14:00:05,627 - distributed.worker - ERROR - argument of type 'NoneType' is not iterable
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 223, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: frames_nbytes = await stream.read_bytes(fmt_size)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 426, in read_bytes
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: self._try_inline_read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 841, in _try_inline_read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: self._check_closed()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/tornado/iostream.py", line 1017, in _check_closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: raise StreamClosedError(real_error=self.error)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: tornado.iostream.StreamClosedError: Stream is closed
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: During handling of the above exception, another exception occurred:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: Traceback (most recent call last):
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 180, in wrapper
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: return await method(self, *args, **kwargs) # type: ignore
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/worker.py", line 1211, in handle_scheduler
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: await self.handle_stream(comm)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/core.py", line 821, in handle_stream
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: msgs = await comm.read()
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 239, in read
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: convert_stream_closed_error(self, e)
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: File "/opt/conda/envs/coiled/lib/python3.10/site-packages/distributed/comm/tcp.py", line 140, in convert_stream_closed_error
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: if "UNKNOWN_CA" in exc.reason:
Aug 11 14:00:05 ip-10-0-6-45 cloud-init[1267]: TypeError: argument of type 'NoneType' is not iterable
(https://cloud.coiled.io/dask-engineering/clusters/48659/details on hendrik-debug-worker-8ede19223f)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
bugSomething is brokenSomething is broken