Project

General

Profile

Actions
Copy link

Bug #63529

closed

Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3

Added by Max Carrara over 2 years ago. Updated 12 months ago.

Status:
Duplicate
Priority:
Normal
Assignee:
-
Category:
ceph-mgr
Target version:
-
% Done:

100%

Source:
Community (dev)
Backport:
reef
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
Ceph - v18.2.0
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

Our users noticed that Ceph's dashboard is broken in Proxmox Virtual Environment 8. On a more closer investigation, this is apparently caused by an import check in Python modules that use PyO3:

Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Traceback (most recent call last):
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/__init__.py", line 60, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .module import Module, StandbyModule  # noqa: F401
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/module.py", line 30, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .controllers import Router, json_error_page
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._api_router import APIRouter
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_api_router.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._router import Router
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_router.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ._base_controller import BaseController
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/controllers/_base_controller.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from ..services.auth import AuthManager, JwtManager
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/usr/share/ceph/mgr/dashboard/services/auth.py", line 12, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     import jwt
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/__init__.py", line 1, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .api_jwk import PyJWK, PyJWKSet
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/api_jwk.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .algorithms import get_default_algorithms
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/algorithms.py", line 6, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from .utils import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/jwt/utils.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.primitives.asymmetric.ec import EllipticCurve
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/primitives/asymmetric/ec.py", line 11, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat._oid import ObjectIdentifier
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:   File "/lib/python3/dist-packages/cryptography/hazmat/_oid.py", line 7, in <module>
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]:     from cryptography.hazmat.bindings._rust import (
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: ImportError: PyO3 modules may only be initialized once per interpreter process
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Class not found in module 'dashboard'
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.438+0200 7fecdc91e000 -1 mgr[py] Error loading module 'dashboard': (2) No such file or directory
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.470+0200 7fecdc91e000 -1 mgr[py] Module progress has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member
Sep 04 18:39:51 ceph-01 ceph-mgr[15669]: 2023-09-04T18:39:51.502+0200 7fecdc91e000 -1 log_channel(cluster) log [ERR] : Failed to load ceph-mgr modules: dashboard
This error doesn't just appear in PVE, but also in:

This is due to the fact that every ceph-mgr module that uses Python modules that use PyO3 to provide bindings to Rust code will raise an ImportError if imported more than once under the presence of multiple Python sub-interpreters. This check is present in PyO3 version 0.17.0 and upwards and was introduced in this pull request: https://github.com/PyO3/pyo3/pull/2523

So, for now it seems that this "only" completely breaks the dashboard; however, other ceph-mgr modules might also be affected in the future, depending on whether they use PyO3 Python modules or not.

This also means that the official distribution of Ceph will be affected sooner or later, either when a newer version of Python's cryptography or PyO3 is used.

The above is essentially a summary of my findings and posts over at the Proxmox forum. For the curious, you can read up on everything starting from this post over here: https://forum.proxmox.com/threads/ceph-warning-post-upgrade-to-v8.129371/post-587100

Related issues 2 (0 open — 2 closed)

Is duplicate of Ceph - Bug #64213: MGR modules incompatible with later PyO3 versions - PyO3 modules may only be initialized once per interpreter processResolvedErnesto Puerta

Actions
Copied to mgr - Backport #66218: reef: Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3ResolvedActions
Actions
Copy link
 #1

Updated by Iggy Jackson about 2 years ago

`bcrypt` is now also doing this, so I think basically every module is broken at this point.

Actions
Copy link
 #3

Updated by Matthew Vernon about 2 years ago

Since that PR is a partial fix for this, I've put in https://github.com/ceph/ceph/pull/55689 to backport it to reef.

Actions
Copy link
 #4

Updated by Hector Martin almost 2 years ago

bcrypt bug: https://github.com/pyca/bcrypt/issues/694

It looks like there is no fix in right, so I think the only solution is for Ceph to stop using the bcrypt module (or start vendoring an unaffected 3.x version somehow).

Note that this is currently breaking Ceph in at least Gentoo and Fedora 40, and presumably every rolling distro and recent releases of many other non-rolling distros by now.

Actions
Copy link
 #5

Updated by Konstantin Shalygin almost 2 years ago

  • Status changed from New to Pending Backport
  • Backport set to reef
  • Pull request ID set to 63529
Actions
Copy link
 #6

Updated by Upkeep Bot almost 2 years ago

  • Copied to Backport #66218: reef: Python Sub-Interpreter Model Used by ceph-mgr Incompatible With Python Modules Based on PyO3 added
Actions
Copy link
 #8

Updated by Max Carrara almost 2 years ago

For reference, I'm currently working on sub-interpreter support for PyO3.

I have no clear idea when it will be finished, but there's been a bit of progress recently, finally.

For anyone that's stumbling over this issue here and wants to see what's going on, the tracking issue at PyO3's is over here: https://github.com/PyO3/pyo3/issues/3451

If you want to help out, let me know over at: https://github.com/Aequitosh/pyo3/discussions/1

Implementing this still has a long way to go, and it will probably take even longer until it trickles downstream to Python packages like bcrypt & cryptography; nevertheless, it's going somewhere.

Actions
Copy link
 #9

Updated by Konstantin Shalygin almost 2 years ago

  • Status changed from Pending Backport to Resolved
  • % Done changed from 0 to 100
Actions
Copy link
 #10

Updated by Hector Martin almost 2 years ago

Why was this closed? This is not fixed. The dependency on cryptography was removed, but the dependency on bcrypt remains, and that is still breaking everything.

Please reopen. This won't be fixed until Ceph either stops depending on external bcrypt or bcrypt/pyo3 starts working with subinterpreters.

Actions
Copy link
 #11

Updated by Nizamudeen A almost 2 years ago

  • Related to Bug #64213: MGR modules incompatible with later PyO3 versions - PyO3 modules may only be initialized once per interpreter process added
Actions
Copy link
 #12

Updated by Nizamudeen A almost 2 years ago

Hector,

The core issue is tracked via a different tracker which I linked here.

Actions
Copy link
 #13

Updated by Iggy Jackson almost 2 years ago

Seems weird to make the newer bug with less info the main bug for this issue, but maybe I just didn't understand how redmine works.

Actions
Copy link
 #14

Updated by Ernesto Puerta 12 months ago

  • Status changed from Resolved to Duplicate
  • Pull request ID deleted (63529)
Actions
Copy link
 #15

Updated by Ernesto Puerta 12 months ago

  • Related to deleted (Bug #64213: MGR modules incompatible with later PyO3 versions - PyO3 modules may only be initialized once per interpreter process)
Actions
Copy link
 #16

Updated by Ernesto Puerta 12 months ago

  • Is duplicate of Bug #64213: MGR modules incompatible with later PyO3 versions - PyO3 modules may only be initialized once per interpreter process added
Actions
Copy link

Also available in: Atom PDF