Potential race condition in GenericProxy.create_ssl_cert. #1811
Description
There appears to be a race condition when multiple threads invoke GenericProxy.create_ssl_cert when using USE_SSL=1. It causes a cascading failure of the service as shown by the following logs.
[36mlocalstack_1 |�[0m 2019-11-27T18:59:49:ERROR:localstack.services.generic_proxy: Exception running proxy on port 4576: [SSL] PEM lib (_ssl.c:3503) Traceback (most recent call last):
�[36mlocalstack_1 |�[0m File "/opt/code/localstack/localstack/services/generic_proxy.py", line 386, in run_cmd
�[36mlocalstack_1 |�[0m server_side=True, certfile=combined_file)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
�[36mlocalstack_1 |�[0m ciphers=ciphers)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 750, in __init__
�[36mlocalstack_1 |�[0m self._context.load_cert_chain(certfile, keyfile)
�[36mlocalstack_1 |�[0m ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
�[36mlocalstack_1 |�[0m
�[36mlocalstack_1 |�[0m 2019-11-27T18:59:49:ERROR:localstack.services.generic_proxy: Exception running proxy on port 4569: [SSL] PEM lib (_ssl.c:3503) Traceback (most recent call last):
�[36mlocalstack_1 |�[0m File "/opt/code/localstack/localstack/services/generic_proxy.py", line 386, in run_cmd
�[36mlocalstack_1 |�[0m server_side=True, certfile=combined_file)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
�[36mlocalstack_1 |�[0m ciphers=ciphers)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 750, in __init__
�[36mlocalstack_1 |�[0m self._context.load_cert_chain(certfile, keyfile)
�[36mlocalstack_1 |�[0m ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
�[36mlocalstack_1 |�[0m
�[36mlocalstack_1 |�[0m 2019-11-27T18:59:49:INFO:localstack.multiserver: Starting multi API server process on port 51492
�[36mlocalstack_1 |�[0m 2019-11-27T18:59:49:ERROR:localstack.services.generic_proxy: Exception running proxy on port 4568: [SSL] PEM lib (_ssl.c:3503) Traceback (most recent call last):
�[36mlocalstack_1 |�[0m File "/opt/code/localstack/localstack/services/generic_proxy.py", line 386, in run_cmd
�[36mlocalstack_1 |�[0m server_side=True, certfile=combined_file)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
�[36mlocalstack_1 |�[0m ciphers=ciphers)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 750, in __init__
�[36mlocalstack_1 |�[0m self._context.load_cert_chain(certfile, keyfile)
�[36mlocalstack_1 |�[0m ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
�[36mlocalstack_1 |�[0m
�[36mlocalstack_1 |�[0m 2019-11-27T18:59:49:ERROR:localstack.services.generic_proxy: Exception running proxy on port 4586: [SSL] PEM lib (_ssl.c:3503) Traceback (most recent call last):
�[36mlocalstack_1 |�[0m File "/opt/code/localstack/localstack/services/generic_proxy.py", line 386, in run_cmd
�[36mlocalstack_1 |�[0m server_side=True, certfile=combined_file)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 1158, in wrap_socket
�[36mlocalstack_1 |�[0m ciphers=ciphers)
�[36mlocalstack_1 |�[0m File "/usr/lib/python3.6/ssl.py", line 750, in __init__
�[36mlocalstack_1 |�[0m self._context.load_cert_chain(certfile, keyfile)
�[36mlocalstack_1 |�[0m ssl.SSLError: [SSL] PEM lib (_ssl.c:3503)
�[36mlocalstack_1 |�[0m
common.generate_ssl_cert has some os.path.exists checks while later multiple independent save_file calls. These look like they might race and cause the 3 ssl files to become invalid.