Add TLS settings for HTTP/2#3456
Conversation
|
@pquentin If you have any questions or suggestions, please do tell! I have tried my best to explain everything, but I will be happy to provide more details if I am unclear anywhere! |
pquentin
left a comment
There was a problem hiding this comment.
Sorry for the delay, I missed that PR in my notifications somehow.
NOTE: PHA is set to False in http2 mode even if it was set to True, which is in line with the previous design which set it to True even if it was set 'False', though I could not find a reason for it (#1635, the same was asked in pr but post-commit). This should be reviewed.
Thanks for the references. Disabling PHA altogether with HTTP/2 is fine.
The final list of cipher suites from this string in OpenSSL 3.0.2 and 1.1.1f:
Would you mind checking with OpenSSL 3.3 too for completeness?
Regarding CI, how do you explain the PyPy failures?
2024-08-19T18:55:15.4166853Z ______________ ERROR at setup of TestHTTPS_TLSv1.test_simple[h11] ______________
2024-08-19T18:55:15.4167725Z Traceback (most recent call last):
2024-08-19T18:55:15.4169470Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/runner.py", line 342, in from_call
2024-08-19T18:55:15.4171201Z result: Optional[TResult] = func()
2024-08-19T18:55:15.4172394Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/runner.py", line 263, in <lambda>
2024-08-19T18:55:15.4173726Z lambda: ihook(item=item, **kwds), when=when, reraise=reraise
2024-08-19T18:55:15.4175503Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
2024-08-19T18:55:15.4176862Z return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2024-08-19T18:55:15.4177914Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
2024-08-19T18:55:15.4179072Z return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2024-08-19T18:55:15.4180241Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
2024-08-19T18:55:15.4181114Z raise exception.with_traceback(exception.__traceback__)
2024-08-19T18:55:15.4182182Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
2024-08-19T18:55:15.4183170Z teardown.throw(exception) # type: ignore[union-attr]
2024-08-19T18:55:15.4184232Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/unraisableexception.py", line 85, in pytest_runtest_setup
2024-08-19T18:55:15.4185693Z yield from unraisable_exception_runtest_hook()
2024-08-19T18:55:15.4187759Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/unraisableexception.py", line 65, in unraisable_exception_runtest_hook
2024-08-19T18:55:15.4189434Z yield
2024-08-19T18:55:15.4190873Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
2024-08-19T18:55:15.4192747Z teardown.throw(exception) # type: ignore[union-attr]
2024-08-19T18:55:15.4194628Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/logging.py", line 833, in pytest_runtest_setup
2024-08-19T18:55:15.4196178Z yield from self._runtest_for(item, "setup")
2024-08-19T18:55:15.4197885Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/logging.py", line 822, in _runtest_for
2024-08-19T18:55:15.4199253Z yield
2024-08-19T18:55:15.4200700Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
2024-08-19T18:55:15.4202379Z teardown.throw(exception) # type: ignore[union-attr]
2024-08-19T18:55:15.4204235Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/capture.py", line 877, in pytest_runtest_setup
2024-08-19T18:55:15.4206139Z return (yield)
2024-08-19T18:55:15.4208052Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
2024-08-19T18:55:15.4209655Z teardown.throw(exception) # type: ignore[union-attr]
2024-08-19T18:55:15.4211283Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/threadexception.py", line 82, in pytest_runtest_setup
2024-08-19T18:55:15.4212615Z yield from thread_exception_runtest_hook()
2024-08-19T18:55:15.4214612Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/threadexception.py", line 63, in thread_exception_runtest_hook
2024-08-19T18:55:15.4215937Z yield
2024-08-19T18:55:15.4217176Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
2024-08-19T18:55:15.4218678Z res = hook_impl.function(*args)
2024-08-19T18:55:15.4220260Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/runner.py", line 158, in pytest_runtest_setup
2024-08-19T18:55:15.4221679Z item.session._setupstate.setup(item)
2024-08-19T18:55:15.4223189Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/runner.py", line 514, in setup
2024-08-19T18:55:15.4224431Z raise exc
2024-08-19T18:55:15.4225724Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/runner.py", line 511, in setup
2024-08-19T18:55:15.4226947Z col.setup()
2024-08-19T18:55:15.4228246Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/python.py", line 1834, in setup
2024-08-19T18:55:15.4229525Z self._request._fillfixtures()
2024-08-19T18:55:15.4231042Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 689, in _fillfixtures
2024-08-19T18:55:15.4232725Z item.funcargs[argname] = self.getfixturevalue(argname)
2024-08-19T18:55:15.4234278Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 547, in getfixturevalue
2024-08-19T18:55:15.4235698Z fixturedef = self._get_active_fixturedef(argname)
2024-08-19T18:55:15.4237459Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 566, in _get_active_fixturedef
2024-08-19T18:55:15.4239011Z self._compute_fixture_value(fixturedef)
2024-08-19T18:55:15.4240620Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 648, in _compute_fixture_value
2024-08-19T18:55:15.4241931Z fixturedef.execute(request=subrequest)
2024-08-19T18:55:15.4243307Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 1087, in execute
2024-08-19T18:55:15.4244665Z result = ihook.pytest_fixture_setup(fixturedef=self, request=request)
2024-08-19T18:55:15.4246182Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_hooks.py", line 513, in __call__
2024-08-19T18:55:15.4247572Z return self._hookexec(self.name, self._hookimpls.copy(), kwargs, firstresult)
2024-08-19T18:55:15.4249096Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_manager.py", line 120, in _hookexec
2024-08-19T18:55:15.4250438Z return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
2024-08-19T18:55:15.4252009Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 139, in _multicall
2024-08-19T18:55:15.4253341Z raise exception.with_traceback(exception.__traceback__)
2024-08-19T18:55:15.4254832Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 122, in _multicall
2024-08-19T18:55:15.4256223Z teardown.throw(exception) # type: ignore[union-attr]
2024-08-19T18:55:15.4258025Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/setuponly.py", line 36, in pytest_fixture_setup
2024-08-19T18:55:15.4259324Z return (yield)
2024-08-19T18:55:15.4260602Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/pluggy/_callers.py", line 103, in _multicall
2024-08-19T18:55:15.4261832Z res = hook_impl.function(*args)
2024-08-19T18:55:15.4263288Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 1140, in pytest_fixture_setup
2024-08-19T18:55:15.4264694Z result = call_fixture_func(fixturefunc, request, kwargs)
2024-08-19T18:55:15.4266270Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/_pytest/fixtures.py", line 917, in call_fixture_func
2024-08-19T18:55:15.4267760Z fixture_result = fixturefunc(**kwargs)
2024-08-19T18:55:15.4268774Z File "/home/runner/work/urllib3/urllib3/test/conftest.py", line 316, in supported_tls_versions
2024-08-19T18:55:15.4269768Z ssl_context=ssl_.create_urllib3_context(
2024-08-19T18:55:15.4271315Z File "/home/runner/work/urllib3/urllib3/.nox/test-pypy3-10/lib/pypy3.10/site-packages/urllib3/util/ssl_.py", line 304, in create_urllib3_context
2024-08-19T18:55:15.4272915Z context.minimum_version = TLSVersion.TLSv1_2
2024-08-19T18:55:15.4273831Z NameError: name 'TLSVersion' is not defined. Did you mean: 'ssl_version'?There are two confusing things here. First, on my laptop I do have ssl.TLSVersion in PyPy (even if CPython deprecated it). Secondly, how could your changes affect this code? create_urllib3_context itself uses the same behavior as before when http is False (the default value).
PyPy also fails on my laptop, but with TypeError: Can't create an SSLContext object without an ssl module while I definitely have an ssl module. 🤔
| def test_create_urllib3_context_http2_default_ciphers( | ||
| self, monkeypatch: pytest.MonkeyPatch | ||
| ) -> None: | ||
| ciphers = ssl_.HTTP2_DEFAULT_CIPHERS | ||
| context = mock.create_autospec(ssl_.SSLContext) | ||
| context.set_ciphers = mock.Mock() | ||
| context.options = 0 | ||
| monkeypatch.setattr(ssl_, "SSLContext", lambda *_, **__: context) | ||
|
|
||
| assert ssl_.create_urllib3_context(http2=True) is context | ||
|
|
||
| assert context.set_ciphers.call_count == 1 | ||
| assert context.set_ciphers.call_args == mock.call(ciphers) |
There was a problem hiding this comment.
What are your thought on rewriting this using get_ciphers() and asserting that the names that you want are here? Such as ECDHE-RSA-AES128-GCM-SHA256.
There was a problem hiding this comment.
I agree, that will be a better idea! But unfortunately pyopenssl does not have get_ciphers method in context class. They have a get_cipher_list but in Connection, which requires a socket, but passing a mock does not seem to work. Maybe there is an alternate way?
There was a problem hiding this comment.
We could try implementing get_ciphers, but I have no idea how hard this is. Don't worry about it then.
|
Thanks for the review! I was looking into the pypy issues. I think the issue arises from the fact that pypy's ssl module does not have the But I am not sure what can we do about it at the moment, what do you think @pquentin, any suggestions? |
|
I guess pypy issue is solved, though I had to modify the import mechanism in |
pquentin
left a comment
There was a problem hiding this comment.
I believe I have answered to all comments.
Done. I did not find any change. I think I have resolved all the review comments. Thanks for reviewing! Also I was wondering maybe we could also remove DHE from cipher suites, considering they are weak and Hardening Your Web Server’s SSL Ciphers also recommends against using them. I had kept it mostly for legacy reasons and I could not come to a conclusion whether to keep them or not. What do you think @pquentin? |
|
Adding some arguments for and against using DHE. For:
Against:
|
pquentin
left a comment
There was a problem hiding this comment.
Let's remove the DHE cipher suites, we can always add them back if that proves to be a problem. The rest looks good to me, thank you!
Done! For future reference, cipher suites selected by OpenSSL(3.3.0, 3.0.2 and 1.1.1f) using Thank you! |
|
Really appreciated working with you on this. Feel free to claim your bounty on Open Collective now! |
(Fixes #3306)
This PR modifies
create_urllib3_contextto add TLS settings required for HTTP/2 as per RFC 9113.Note that the changes are made such that we default to settings mandated by RFC 9113 but a user/client may use their configuration which may not comply with RFC 9113.
TLS settings mandated by RFC 9113 in section 9.2 and its compliance in urllib3 after this PR:
(Sections 9.2.1 and 9.2.2 are for TLSv1.2 and 9.2.3 is for TLSv1.3)
@SECLEVEL=2:ECDHE+AESGCM:ECDHE+CHACHA20:!aDSS(EDIT: Updated it new)The initial handshake is done by SSL on the creation of a socket, but the mechanism for post-handshake authentication(PHA) is left to
sslusers, though I may be incorrect. Since h2 does not handle this, PHA is disabled in http2 mode. EDIT: Some additional context, Firefox disables PHA by default and Chrome does not support it.NOTE: PHA is set to False in http2 mode even if it was set to True, which is in line with the previous design which set it to True even if it was set 'False', though I could not find a reason for it (#1635, the same was asked in pr but post-commit). This should be reviewed.
TLS cipher suites with HTTP/2
Here are a few points that are important in choosing the cipher suites:
RFC 9113 has a list of prohibited ciphers, which I should mention is the same as in RFC 7540. It also mentions at the end of the blocklist(Appendix A) that the list includes cipher suites with non-ephemeral key exchange or ones based on TLS null, stream, or block cipher type. Thus, only cipher suites that use a ephemeral key exchange and are NOT based on the TLS null, stream, or block cipher are allowed.
RFC 9113 9.2.1 mandates "ephemeral key exchange sizes of at least 2048 bits for cipher suites that use ephemeral finite field Diffie-Hellman (DHE) and 224 bits for cipher suites that use ephemeral elliptic curve Diffie-Hellman (ECDHE)". This corresponds to Security Level 2 or "@SECLEVEL=2".
RFC 9113 9.2.2 mandates support for TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256.
I also referred to Hardening Your Web Server’s SSL Ciphers and
ssl._DEFAULT_CIPHERS. But both were not adequate for RFC 9113 requirements and allowed at least 2 cipher suites from the blocklist when verified using OpenSSL 3.0.2.The final selection is equivalent to the one used in python-h2 examples, see Negotiating HTTP/2](http://python-hyper.org/projects/h2/en/stable/negotiating-http2.html), with some extra flags was picked from
ssl, though they do not affect the final set of ciphers which is selected when checked with OpenSSL 3.0.2 and OpenSSL 1.1.1f. Also, the final set of cipher suites are the same as the ones recommended by Mozilla for general-purpose servers(which implies most of clients).The cipher suites that is used:
@SECLEVEL=2:ECDHE+AESGCM:ECDHE+CHACHA20:DHE+AESGCM:DHE+CHACHA20:!aNULL:!eNULL:!aDSS:!SHA1@SECLEVEL=2: security level 2 with 112 bits minimum securityECDHEandDHE: both of which are ephemeral key exchanges.AESGCMandCHACHA20!aNULL:!eNULL: no null ciphers!aDSS: no authentication with discrete logarithm DSA algorithm!SHA1: no weak SHA1 MACThe final list of cipher suites from this string in OpenSSL 3.0.2 and 1.1.1f(EDIT: also checked against OpenSSL 3.3.0):
Note that this contains TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(OpenSSL name: ECDHE-RSA-AES128-GCM-SHA256) which is required.