Ceph

Many MGR modules cannot be used on platforms with later versions of PyO3. The error message

PyO3 modules may only be initialized once per interpreter process

is issued in many places, including (but not limited to) the dashboard, and any TLS communication.

This occurs in Debian 12 (bookworm), but has been noted in other distributions too.

An example crash is:

$ ceph crash info 2024-01-12T11:10:03.938478Z_2263d2c8-8120-417e-84bc-bb01f5d81e52
{
    "backtrace": [
        "  File \"/usr/share/ceph/mgr/cephadm/__init__.py\", line 1, in <module>\n    from .module import CephadmOrchestrator",
        "  File \"/usr/share/ceph/mgr/cephadm/module.py\", line 15, in <module>\n    from cephadm.service_discovery import ServiceDiscovery",
        "  File \"/usr/share/ceph/mgr/cephadm/service_discovery.py\", line 20, in <module>\n    from cephadm.ssl_cert_utils import SSLCerts",
        "  File \"/usr/share/ceph/mgr/cephadm/ssl_cert_utils.py\", line 8, in <module>\n    from cryptography import x509",
        "  File \"/lib/python3/dist-packages/cryptography/x509/__init__.py\", line 6, in <module>\n    from cryptography.x509 import certificate_transparency",
        "  File \"/lib/python3/dist-packages/cryptography/x509/certificate_transparency.py\", line 10, in <module>\n    from cryptography.hazmat.bindings._rust import x509 as rust_x509",
        "ImportError: PyO3 modules may only be initialized once per interpreter process" 
    ],
    "ceph_version": "18.2.1",
    "crash_id": "2024-01-12T11:10:03.938478Z_2263d2c8-8120-417e-84bc-bb01f5d81e52",
    "entity_name": "mgr.xxxxx01",
    "mgr_module": "cephadm",
    "mgr_module_caller": "PyModule::load_subclass_of",
    "mgr_python_exception": "ImportError",
    "os_id": "12",
    "os_name": "Debian GNU/Linux 12 (bookworm)",
    "os_version": "12 (bookworm)",
    "os_version_id": "12",
    "process_name": "ceph-mgr",
    "stack_sig": "7815ad73ced094695056319d1241bf7847da19b4b0dfee7a216407b59a7e3d84",
    "timestamp": "2024-01-12T11:10:03.938478Z",
    "utsname_hostname": "xxxxx01.xxx.xxx",
    "utsname_machine": "x86_64",
    "utsname_release": "6.1.0-17-amd64",
    "utsname_sysname": "Linux",
    "utsname_version": "#1 SMP PREEMPT_DYNAMIC Debian 6.1.69-1 (2023-12-30)" 
}

My understanding of the relevant background is:

• MGR modules use python subinterpreters for isolation between modules.
• Several modules (including but not limited to dashboard & restful) use python3-cryptography for hashing and TLS (and possibly other things).
• python3-cryptography delegates some crypto functions to Rust functions. These include bcrypt and TLS-related functions.
• python3-cryptography uses PyO3 to invoke Rust functions.
• PyO3 does not support being used by subinterpreters. In the past this has been allowed but was actually unsafe. Now PyO3 throws an exception when it detects multiple initialisations.

So it appears that the MGR use of these functions has always been unsafe, and is now forbidden.

PR54710 identified that the code necessary for the bcrypt hashing used during authentication could easily be written in a small amount of native python, thus avoiding the whole PyO3 area altogether.

However there was a note in the discussions that you also had to disable TLS. And it only applied to the dashboard. My stacktrace above shows the exception during TLS initialisation.

As PyO3 updates are adopted in other linux distributions and containers this is likely to break a number of MGR modules. As there does not seem to be any subinterpreter support in PyO3 coming soon, the only option may be to completely eliminate use of python3-cryptopgraphy from all MGR modules. (It is possible MGR modules may also use other python3 modules that use PyO3 to invoke Rust).

Unfortunately for us, we didn't find this until we had upgraded all MONs in a cluster to reef, at which point we can't downgrade them to quincy. And we can't upgrade the MGR. As a temporary measure (this cluster had MON/MGR/MDS/RGW colocated on 2 hosts) we have added another bookworm host running a reef MON to ensure we can maintain quorum. We are not sure whether it is safe to upgrade the other components (OSD, MDS, RGW) while the MGR remains at quincy,

This has been discussed on ceph-users, and the postings contain links to several other sources of information. Only Problem 2 referred to on that thread is relevant to this bug:

https://lists.ceph.io/hyperkitty/list/ceph-users@ceph.io/thread/VEN3IU53ZVU343S3U25QKKPCOER4X7AG/#53P4RTSCPPHQYIEISBBAEJXQJNEQSWYL

Copied from the issue #63529:

Our users noticed that Ceph's dashboard is broken in Proxmox Virtual Environment 8. On a more closer investigation, this is apparently caused by an import check in Python modules that use PyO3:

> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Traceback (most recent call last):
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .module import Module, StandbyModule  # noqa: F401
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 30, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .controllers import Router, json_error_page
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 1, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._api_router import APIRouter
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_api_router.py", line 1, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._router import Router
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_router.py", line 7, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._base_controller import BaseController
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 11, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ..services.auth import AuthManager, JwtManager
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/services/auth.py", line 12, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     import jwt
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/__init__.py", line 1, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .api_jwk import PyJWK, PyJWKSet
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/api_jwk.py", line 6, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .algorithms import get_default_algorithms
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/algorithms.py", line 6, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .utils import (
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/utils.py", line 7, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat._oid import ObjectIdentifier
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/_oid.py", line 7, in <module>
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.bindings._rust import (
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: ImportError: PyO3 modules may only be initialized once per interpreter process
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Class not found in module 'dashboard'
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.470+0200 7fecdc91e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
> Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard
> 

This error doesn't just appear in PVE, but also in:

• Debian 12 (Bookworm), see: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1055212
• ArchLinux (AUR), see: https://github.com/bazaah/aur-ceph/issues/20

This is due to the fact that every ceph-mgr module that uses Python modules that use PyO3 to provide bindings to Rust code will raise an ImportError if imported more than once under the presence of multiple Python sub-interpreters. This check is present in PyO3 version 0.17.0 and upwards and was introduced in this pull request: https://github.com/PyO3/pyo3/pull/2523

So, for now it seems that this "only" completely breaks the dashboard; however, other ceph-mgr modules might also be affected in the future, depending on whether they use PyO3 Python modules or not.

This also means that the official distribution of Ceph will be affected sooner or later, either when a newer version of Python's cryptography or PyO3 is used.

The above is essentially a summary of my findings and posts over at the Proxmox forum. For the curious, you can read up on everything starting from this post over here: https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/post-587100