tentacle: mgr:python: avoid pyo3 errors by running certain cryptographic functions in a child process#66794
tentacle: mgr:python: avoid pyo3 errors by running certain cryptographic functions in a child process#66794nizamial09 merged 25 commits intotentaclefrom
Conversation
| self.fail('Invalid private key: %s' % str(e)) | ||
| _crt = self._load_cert(crt) | ||
| try: | ||
| context = SSL.Context(SSL.TLSv1_METHOD) |
Check failure
Code scanning / CodeQL
Use of insecure SSL/TLS version High
There was a problem hiding this comment.
While reviewing this (again 😅 ) I realized that we're still using PyOpenSSL... Based on the warning these folks posted:
Note
Note: The Python Cryptographic Authority strongly suggests the use of pyca/cryptography where possible. If you are using pyOpenSSL for anything other than making a TLS connection you should move to cryptography and drop your pyOpenSSL dependency.
context = SSL.Context(SSL.TLSv1_METHOD)could be easily replace with CPython's built-in ssl solution:
context = ssl.create_default_context() # TLS >=1.2Same for Private Key/Certificate generation:
def create_private_key(self) -> str:
pkey = crypto.PKey()
pkey.generate_key(crypto.TYPE_RSA, 2048)
return crypto.dump_privatekey(crypto.FILETYPE_PEM, pkey).decode()into
from cryptography.hazmat.primitives.asymmetric import rsa
from cryptography.hazmat.primitives import serialization
def create_private_key(self) -> str:
key = rsa.generate_private_key(
public_exponent=65537,
key_size=2048,
)
pem = key.private_bytes(
encoding=serialization.Encoding.PEM,
format=serialization.PrivateFormat.TraditionalOpenSSL,
encryption_algorithm=serialization.NoEncryption(),
)
return pem.decode("utf-8")There was a problem hiding this comment.
Given that we have to "suffer" from the cryptography PyO3 side-effects, let's fully use it 😅
There was a problem hiding this comment.
sounds good. but this needs to come in from the main branch to the tentacle so I'll open a separate PR for it and test it out..
…itialized once per interpreter process' issue. Fixes: https://tracker.ceph.com/issues/64213 Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit 717d0a6)
Where possible try to use structured output in JSON for easier parsing and interaction with the parent process. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 6f2d92c)
Create a class to act as a common shim between the cryptotools external functions and the mgr. It provides common conversion mechanisms and could possibly act as an abstraction in case we decide to make the external function calls in different ways in the future. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 84710f9)
Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit a5861c1)
Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 9dcfde7)
Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 2b9cf24)
Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit 4bcab13)
…he exit. Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit 56d508f)
Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit 218d84f)
The remote verify_tls function was not raising errors when it should. Fix the function so that it always returns an object when it succeeds or fails gracefully. Always parse that function in the crypto caller class. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 96a7a72)
…ode/decode. Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit e364df3)
Signed-off-by: Paulo E. Castro <pecastro@wormholenet.com> (cherry picked from commit 21d6e1d)
Name the parser objects after their functions and not `foo` and `bar`. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 5d4eeff)
Replace a direct usage of bycrypt with our cryptocaller wrapper. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit d2fd81e)
Why violate the typing in a test? mypy never noticed this because tests are not type checked but there seems to be no need to turn a str into bytes to pass to a function that is typed only as taking str! Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 955143d)
The functions now handle the i/o but allow the crypto function class to centralize the functions that actually use the crypto libs. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 4e4cfa5)
Use a main function to encapsulate the cli parsing rather than a block of code in module scope. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit c98e53f)
Lightly reorganize and make the "endpoint" functions in cryptotools.py more consistent and uniform. Use small functions for input and output handling so that the handling is done the same way throughout. Pass a pre-constructed crypto caller via the args to then endpoint functions. Make generating the private key it's own named function rather than one single (and only) function with overloaded behavior controlled by a cli switch. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 552d7b4)
Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit c3dc34a)
Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 0c774d5)
Add a module to select a desired crypto caller. Update the callers to use the crypto caller interface. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 0eb2f4b)
Previously, the internal crypto caller would catch (and convert) some errors when reading the cert but not all cases. Move the logic to catch the errors to a common location and do it once consistently. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit f6ab087)
The cephadm modules needs to use python cryptography module for ssh (via asyncssh) and thus there's no need to use the remote crypto caller in cephadm. Configure cephadm to always use the internal cryptocaller. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 2128ffa)
Add a mgr config option `crypto_caller` that lets a ceph user override the default behavior of using the remote crypto caller. Supported values are `internal` and `remote`. Signed-off-by: John Mulligan <jmulligan@redhat.com> (cherry picked from commit 27c2050)
1dd8abe to
6f45a56
Compare
|
jenkins test windows |
|
@nizamial09 this would need teuthology and also ensuring ceph dash and adm are not releated |
|
@afreen23 yup. I'll wait for cephadm teams approval and probably @adk3798 or @phlogistonjohn can help me schedule a run. |
|
I'll also look athe dashboard e2e. I remember we need to fix something in main related to pyvenv. Probably we need to backport that too |
in frontend e2e.sh file, we don't need to start the node venv early on before the ceph cluster is started. we only need it for the `npm` or `npx` commands. Starting node virtual env and then starting ceph will cause the ceph cluster to assume the node-env python as the python environment which breaks the cryptotools call. So moving the node-env venv start after the ceph is created Fixes: https://tracker.ceph.com/issues/73804 Signed-off-by: Nizamudeen A <nia@redhat.com> (cherry picked from commit a56ae5b)
done, backported #66143 |
|
jenkins test api |
|
jenkins test dashboard |
|
jenkins test windows |
|
cephadm e2e failure will need a backport of a different PR but I'll do that separately. Let's see if the dashboard e2e passes here. |
|
merging it now since tentacle is failing for a while |
Backports upstream fix from ceph/ceph#66794 which runs cryptographic functions in a child process to avoid the conflict.
backport of #62951, #66143