python 3.6 TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket #632
Description
recently in the openstack ci we started to see the following error
TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket
https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/screen-n-novnc-cell1.txt#29
this seams similar to #201
there has not been a new eventlet release or websockify release in several months so its not clear why this would start failing now. we have not actully debuged this much but i was wondering if this seam familar or is a known issue?
what is alittle bit confusing is we have a passing build from 4 days ago https://zuul.opendev.org/t/openstack/build/1d23dbdf6fa84702ab543602cae1ba67 which seams to be useing more or less the same versions
as the failing build https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79
the eventlet version used was 0.25.2
https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/pip3-freeze.txt#31
we were also using openssl version 1.1.1-1ubuntu2.1~18.04.6 in both builds.
comparing the python package difference between the two runs there are some but nothing that looks related
http://paste.openstack.org/show/796119/
and it looks like we were using the same python build python 3.6.7-1~18.04 in both cases
https://zuul.opendev.org/t/openstack/build/1180d2313ade4d5480550313bcd20b79/log/controller/logs/devstacklog.txt#656
https://zuul.opendev.org/t/openstack/build/1d23dbdf6fa84702ab543602cae1ba67/log/controller/logs/devstacklog.txt#656
so at first glance its not clear why one would pass and the the other would fail which is why i tought i would ask the experts.
----------------error traceback -------------------
Jul 20 10:44:42.425350 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: INFO nova.console.websocketproxy [None req-4c318998-3a74-4929-beab-b774a72844cf None None] handler exception: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: DEBUG nova.console.websocketproxy [None req-4c318998-3a74-4929-beab-b774a72844cf None None] exception {{(pid=23498) vmsg /usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py:634}}
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy Traceback (most recent call last):
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 691, in top_new_client
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 619, in do_handshake
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self)
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 100, in __init__
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 99, in __init__
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle()
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 315, in handle
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.handle(self)
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/http/server.py", line 418, in handle
Jul 20 10:44:42.435706 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_one_request()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 47, in handle_one_request
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy super(WebSocketRequestHandlerMixIn, self).handle_one_request()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy method()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 60, in _websocket_do_GET
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_upgrade()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 221, in handle_upgrade
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy WebSocketRequestHandlerMixIn.handle_upgrade(self)
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 87, in handle_upgrade
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_websocket()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 259, in handle_websocket
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.new_websocket_client()
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 265, in new_websocket_client
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy tsock = self.server.security_proxy.connect(tenant_sock, tsock)
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/securityproxy/rfb.py", line 181, in connect
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy compute_sock = scheme.security_handshake(compute_sock)
Jul 20 10:44:42.437262 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/rfb/authvencrypt.py", line 143, in security_handshake
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy ca_certs=CONF.vnc.vencrypt_ca_certs)
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/eventlet/green/ssl.py", line 414, in wrap_socket
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy return GreenSSLSocket(sock, *a, **kw)
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/eventlet/green/ssl.py", line 106, in __init__
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy ca_certs, do_handshake_on_connect and six.PY2, *args, **kw)
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/ssl.py", line 809, in __init__
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy server_hostname)
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy TypeError: _wrap_socket() argument 1 must be _socket.socket, not GreenSSLSocket
Jul 20 10:44:42.439249 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy
Jul 20 10:44:42.593330 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: DEBUG nova.console.websocketproxy [-] 10.209.97.2: new handler Process {{(pid=22231) vmsg /usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py:634}}
Jul 20 10:44:42.602717 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: INFO nova.console.websocketproxy [-] 10.209.97.2 - - [20/Jul/2020 10:44:42] 10.209.97.2: Plain non-SSL (ws://) WebSocket connection
Jul 20 10:44:42.603054 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: INFO nova.console.websocketproxy [-] 10.209.97.2 - - [20/Jul/2020 10:44:42] 10.209.97.2: Path: '/?token=badf437d8bd-d375-435b-a177-5da80474f9db'
Jul 20 10:44:42.652889 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: DEBUG oslo_db.sqlalchemy.engines [None req-550fdb22-9264-4636-b768-401491dbf76c None None] MySQL server mode set to STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION {{(pid=23499) _check_effective_sql_mode /usr/local/lib/python3.6/dist-packages/oslo_db/sqlalchemy/engines.py:304}}
Jul 20 10:44:43.005343 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: DEBUG nova.objects.console_auth_token [None req-550fdb22-9264-4636-b768-401491dbf76c None None] Token validation failed {{(pid=23499) validate /opt/stack/nova/nova/objects/console_auth_token.py:157}}
Jul 20 10:44:43.005925 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: INFO nova.console.websocketproxy [None req-550fdb22-9264-4636-b768-401491dbf76c None None] handler exception: The token '***' is invalid or has expired
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: DEBUG nova.console.websocketproxy [None req-550fdb22-9264-4636-b768-401491dbf76c None None] exception {{(pid=23499) vmsg /usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py:634}}
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy Traceback (most recent call last):
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 691, in top_new_client
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy client = self.do_handshake(startsock, address)
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 619, in do_handshake
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.RequestHandlerClass(retsock, address, self)
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 100, in __init__
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy websockify.ProxyRequestHandler.__init__(self, *args, **kwargs)
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 99, in __init__
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.__init__(self, req, addr, server)
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/socketserver.py", line 724, in __init__
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle()
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 315, in handle
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy SimpleHTTPRequestHandler.handle(self)
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/http/server.py", line 418, in handle
Jul 20 10:44:43.010423 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_one_request()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 47, in handle_one_request
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy super(WebSocketRequestHandlerMixIn, self).handle_one_request()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/lib/python3.6/http/server.py", line 406, in handle_one_request
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy method()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 60, in _websocket_do_GET
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_upgrade()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 221, in handle_upgrade
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy WebSocketRequestHandlerMixIn.handle_upgrade(self)
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websocketserver.py", line 87, in handle_upgrade
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.handle_websocket()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/websockify/websockifyserver.py", line 259, in handle_websocket
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy self.new_websocket_client()
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 197, in new_websocket_client
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy connect_info = self._get_connect_info(ctxt, token)
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/console/websocketproxy.py", line 146, in _get_connect_info
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy connect_info = objects.ConsoleAuthToken.validate(ctxt, token)
Jul 20 10:44:43.011712 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/usr/local/lib/python3.6/dist-packages/oslo_versionedobjects/base.py", line 184, in wrapper
Jul 20 10:44:43.013428 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy result = fn(cls, context, *args, **kwargs)
Jul 20 10:44:43.013428 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy File "/opt/stack/nova/nova/objects/console_auth_token.py", line 158, in validate
Jul 20 10:44:43.013428 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy raise exception.InvalidToken(token='***')
Jul 20 10:44:43.013428 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy nova.exception.InvalidToken: The token '***' is invalid or has expired
Jul 20 10:44:43.013428 ubuntu-bionic-rax-ord-0018472470 nova-novncproxy[22231]: ERROR nova.console.websocketproxy
(temoto) update: format