One 'bad' Transport thread hangs indefinitely at shutdown when multiple Transports are active #520
Description
[MAINTAINER NOTE: a variant of this issue under Python 3 has been worked around as per this comment but it's presumably still at large for Python 2 in some scenarios.]
When running SSH connections to multiple devices the script will sometimes hang once the sys.exit(0) is reached. Once this is reached the script will hang indefinitely. It doesn't happen everytime but the more devices the script runs against the more likely it is to happen. It seems like it's related to the amount of time the script takes to run.
The last log message paramiko outputs is DEBUG:EOF in transport thread
Using the faulthandler module I dumped a stacktrace for when it hangs:
Fatal Python error: Aborted
Thread 0x00007f47faff5700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f47fbff7700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f47fcff9700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f47fdffb700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f47feffd700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f480590f700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f48050ce700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Thread 0x00007f4807368700 (most recent call first):
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 204 in read_all
File "build/bdist.linux-x86_64/egg/paramiko/packet.py", line 341 in read_message
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1590 in run
File "/usr/local/lib/python2.7/threading.py", line 810 in __bootstrap_inner
File "/usr/local/lib/python2.7/threading.py", line 783 in __bootstrap
Current thread 0x00007f480f7c4700 (most recent call first):
File "/usr/local/lib/python2.7/threading.py", line 358 in wait
File "/usr/local/lib/python2.7/threading.py", line 960 in join
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 1419 in stop_thread
File "build/bdist.linux-x86_64/egg/paramiko/transport.py", line 558 in close
File "build/bdist.linux-x86_64/egg/paramiko/resource.py", line 60 in callback
Aborted
It looks like all but LAST ONE of the resources closes correctly but the LAST ONE doesn't close correctly. The same thing happens if .close() is explicitly called on the Transport object. It will hang at that .close() method indefinitely, sometimes.
There is not one specific device it happens for either. I have tried many hosts and OS's with no specific one standing out a problem.