mgr: isolated CherryPy to prevent global state sharing by nizamial09 · Pull Request #67227 · ceph/ceph

nizamial09 · 2026-02-05T10:51:04Z

as the modules are now being loaded onto the main interpreter (see #66244), the
cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree.

So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app.

based on https://docs.cherrypy.dev/en/latest/pkg/cherrypy.process.servers.html#multiple-servers-ports

Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins test classic perf Jenkins Job | Jenkins Job Definition
jenkins test crimson perf Jenkins Job | Jenkins Job Definition
jenkins test signed Jenkins Job | Jenkins Job Definition
jenkins test make check Jenkins Job | Jenkins Job Definition
jenkins test make check arm64 Jenkins Job | Jenkins Job Definition
jenkins test submodules Jenkins Job | Jenkins Job Definition
jenkins test dashboard Jenkins Job | Jenkins Job Definition
jenkins test dashboard cephadm Jenkins Job | Jenkins Job Definition
jenkins test api Jenkins Job | Jenkins Job Definition
jenkins test docs ReadTheDocs | Github Workflow Definition
jenkins test ceph-volume all Jenkins Jobs | Jenkins Jobs Definition
jenkins test windows Jenkins Job | Jenkins Job Definition
jenkins test rook e2e Jenkins Job | Jenkins Job Definition

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

nizamial09 · 2026-02-05T10:54:00Z

I am opening this for some early reviews. Meanwhile I am testing this locally in different environments to see it works without any error.

And it touches some code that I am not entirely familiar (cephadm agent and node-proxy agent servers along with service_discovery) but I was able to verify that they were working in my local kcli environment atleast. I'll refactor that code a bit more (cc: @adk3798 @rkachach )

@NitzanMordhai could we trigger a teuthology along with the other rocky10 PRs to see if this works in teuthology (who knows what issues it can catch other issues with the change 😅

NitzanMordhai · 2026-02-05T11:41:14Z

@NitzanMordhai could we trigger a teuthology along with the other rocky10 PRs to see if this works in teuthology (who knows it can catch other issues with the change 😅

we need to include that PR on next build of branch-of-the-day, need to update the list of PRs and add that one

Copilot

Pull request overview

This PR aims to isolate CherryPy instances across different Ceph manager modules to prevent global state conflicts when modules are loaded in the main interpreter. It introduces a new CherryPyMgr utility class that creates independent WSGI server instances for each module.

Changes:

Introduces cherrypy_module.py with the CherryPyMgr class to manage isolated CherryPy server instances
Refactors prometheus, dashboard, and cephadm modules to use the new isolated server approach
Updates configuration handling to use application-level configs instead of global CherryPy configs

Reviewed changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 13 comments.

Show a summary per file

File	Description
src/pybind/mgr/cherrypy_module.py	New utility class to create isolated CherryPy WSGI server instances
src/pybind/mgr/prometheus/module.py	Refactored to use CherryPyMgr for isolated server instances; config moved from global to app-level
src/pybind/mgr/dashboard/module.py	Updated to use CherryPyMgr; config handling moved to app-level
src/pybind/mgr/dashboard/tools.py	Updated CORS configuration to work with app-level configs
src/pybind/mgr/dashboard/services/auth/auth.py	Removed global CherryPy config updates
src/pybind/mgr/dashboard/tests/init.py	Updated test setup to use isolated tree
src/pybind/mgr/cephadm/module.py	Added debug log message for HTTP server startup
src/pybind/mgr/cephadm/http_server.py	Refactored to start isolated servers for service discovery and agent
src/pybind/mgr/cephadm/agent.py	Removed Server inheritance; updated to work with isolated servers
src/pybind/mgr/cephadm/services/service_discovery.py	Removed Server inheritance; updated to work with isolated servers
debian/ceph-mgr.install	Added cherrypy_module to installation
ceph.spec.in	Added cherrypy_module to RPM spec

Comments suppressed due to low confidence (3)

src/pybind/mgr/cephadm/http_server.py:55

The configure method at line 52 calls self.agent.configure() without any arguments, but the agent.configure method signature was changed to require a tree parameter. This will cause a TypeError when configure() is called. The configure method should be removed or updated to pass the tree argument.

    def configure(self) -> None:
        self.agent.configure()
        self.service_discovery.configure(self.mgr.service_discovery_port,
                                         self.mgr.get_mgr_ip(),
                                         self.security_enabled)

src/pybind/mgr/dashboard/module.py:520

Call to function configure_cors with too few arguments; should be no fewer than 1.

        configure_cors()

src/pybind/mgr/dashboard/module.py:538

Call to function configure_cors with too few arguments; should be no fewer than 1.

        configure_cors()

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/pybind/mgr/prometheus/module.py

src/pybind/mgr/cephadm/services/service_discovery.py

src/pybind/mgr/dashboard/module.py

src/pybind/mgr/cephadm/agent.py

src/pybind/mgr/prometheus/module.py

src/pybind/mgr/cephadm/http_server.py

src/pybind/mgr/prometheus/module.py

src/pybind/mgr/cephadm/http_server.py

src/pybind/mgr/dashboard/module.py

athanatos

Seems broadly reasonable from my understanding. Someone more familiar with these users should give an actual review.

src/pybind/mgr/cherrypy_mgr.py

athanatos · 2026-02-05T20:05:10Z

https://jenkins.ceph.com/job/ceph-dev-pipeline/2821/pipeline-overview/?selected-node=1087

libtcmalloc.so.4()(64bit) libudev.so.1()(64bit) libupb.so.24()(64bit) libz.so.1()(64bit) rtld(GNU_HASH)
Processing files: ceph-mgr-20.3.0-5168.gc852aade.el9.x86_64
error: File not found: /ceph/rpmbuild/BUILDROOT/ceph-20.3.0-5168.gc852aade.el9.x86_64/usr/share/ceph/mgr/cherrypy_module.*
RPM build errors:
    File not found: /ceph/rpmbuild/BUILDROOT/ceph-20.3.0-5168.gc852aade.el9.x86_64/usr/share/ceph/mgr/cherrypy_module.*
2026-02-05 19:13:35,046: INFO: step done: rpm failed in 00:23:30
2026-02-05 19:13:35,046: ERROR: Command failed: Command '['podman', 'run', '--name=ceph_build', '--rm', '--pids-limit=-1', '--user=0', '--env-file=/home/jenkins-build/build/workspace/ceph-dev-pipeline/.env', '--volume=/home/jenkins-build/build/workspace/ceph-dev-pipeline/dist/ceph:/ceph:Z', '-eHOMEDIR=/ceph', 'quay.ceph.io/ceph-ci/ceph-build:c852aad.wip-rocky10-branch-of-the-day-2026-02-05-1770316030-with-cherrypy-pr.centos9.x86_64.default', 'bash', '-c', "set -x; mkdir -p /ceph/rpmbuild && rpmbuild --rebuild '-D_topdir /ceph/rpmbuild' --with=sccache --without=dwz --with=tcmalloc /ceph/ceph-20.3.0-5168.gc852aade.el9.src.rpm"]' returned non-zero exit status 1.
2026-02-05 19:13:35,046: WARNING: 🚧 the command may have faild due to circumstances beyond the influence of this build script. For example: a complier error caused by a source code change. Pay careful attention to the output generated by the command before reporting this as a problem with the build-with-container.py script. 🚧
script returned exit code 1

error: File not found: /ceph/rpmbuild/BUILDROOT/ceph-20.3.0-5168.gc852aade.el9.x86_64/usr/share/ceph/mgr/cherrypy_module.*

Build failure ^

nizamial09 · 2026-02-06T04:17:57Z


Build failure ^

fixed that. I'll retrigger an isolated build to make sure builds are passing. once its passing I'll let you guys know and we can add it along with the rocky10 patches.

Copilot

Pull request overview

Copilot reviewed 15 out of 15 changed files in this pull request and generated 11 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

src/pybind/mgr/dashboard/module.py

src/pybind/mgr/dashboard/tools.py

src/pybind/mgr/cephadm/http_server.py

src/pybind/mgr/cherrypy_mgr.py

src/pybind/mgr/cephadm/services/service_discovery.py

src/pybind/mgr/cherrypy_mgr.py

src/pybind/mgr/dashboard/module.py

nizamial09 · 2026-02-11T15:24:01Z

added unit tests as well.

nizamial09 · 2026-02-12T06:05:43Z

jenkins test make check arm64

src/pybind/mgr/cephadm/services/service_discovery.py

src/pybind/mgr/cephadm/http_server.py

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 8 comments.

src/pybind/mgr/tests/test_cherrypy_mgr.py

src/pybind/mgr/cherrypy_mgr.py

src/pybind/mgr/dashboard/tools.py

src/pybind/mgr/prometheus/module.py

src/pybind/mgr/tests/test_cherrypy_mgr.py

rkachach

LGTM

as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543, https://tracker.ceph.com/issues/74980 Signed-off-by: Nizamudeen A <nia@redhat.com>

ljflores · 2026-02-26T20:38:10Z

@nizamial09 PTAL at this ticket that came up in the Rocky10 testing: https://tracker.ceph.com/issues/75213

nizamial09 · 2026-02-27T02:43:20Z

thanks @ljflores, I also saw that in a different rados suite here and fixed that issue. I was checking the latest rados result which included an updated commit and I don't see the issue there anymore.

ljflores · 2026-03-03T16:02:59Z

thanks @ljflores, I also saw that in a different rados suite here and fixed that issue. I was checking the latest rados result which included an updated commit and I don't see the issue there anymore.

Thanks for confirming!

nizamial09 requested review from NitzanMordhai, athanatos and epuertat February 5, 2026 10:51

nizamial09 added the dashboard label Feb 5, 2026

nizamial09 requested review from a team as code owners February 5, 2026 10:51

github-project-automation bot added this to Ceph-Dashboard Feb 5, 2026

nizamial09 requested a review from aaSharma14 February 5, 2026 10:51

github-project-automation bot moved this to New in Ceph-Dashboard Feb 5, 2026

github-actions bot added cephadm monitoring pybind labels Feb 5, 2026

nizamial09 force-pushed the isolate-cherrypy branch 2 times, most recently from e9c1f37 to e8c450d Compare February 5, 2026 11:01

github-actions bot added the build/ops label Feb 5, 2026

nizamial09 changed the title ~~[wip]: mgr: isolated CherryPy to prevent global state sharing~~ [wip] mgr: isolated CherryPy to prevent global state sharing Feb 5, 2026

epuertat requested a review from Copilot February 5, 2026 12:01

Copilot started reviewing on behalf of epuertat February 5, 2026 12:02 View session

Copilot AI reviewed Feb 5, 2026

View reviewed changes

athanatos reviewed Feb 5, 2026

View reviewed changes

src/pybind/mgr/cherrypy_mgr.py Show resolved Hide resolved

nizamial09 force-pushed the isolate-cherrypy branch from e8c450d to 2285e49 Compare February 6, 2026 04:17

github-actions bot added CI Continuous Integration mgr labels Feb 6, 2026

nizamial09 force-pushed the isolate-cherrypy branch 3 times, most recently from a301de0 to 9c5bf62 Compare February 6, 2026 06:37

epuertat requested a review from Copilot February 11, 2026 09:38

Copilot started reviewing on behalf of epuertat February 11, 2026 09:38 View session

Copilot AI reviewed Feb 11, 2026

View reviewed changes

nizamial09 force-pushed the isolate-cherrypy branch 2 times, most recently from c4ae8e9 to c0f77e4 Compare February 11, 2026 15:22

yaarith added the rocky10 label Feb 12, 2026

rkachach reviewed Feb 16, 2026

View reviewed changes

epuertat requested a review from Copilot February 19, 2026 08:43

Copilot started reviewing on behalf of epuertat February 19, 2026 08:43 View session

Copilot AI reviewed Feb 19, 2026

View reviewed changes

idryomov mentioned this pull request Feb 22, 2026

pybind/rbd: fix Python 3.12+ sub-interpreter crashes in callbacks #66446

Open

14 tasks

nizamial09 force-pushed the isolate-cherrypy branch 2 times, most recently from ee8d2a0 to 0b70cae Compare February 24, 2026 06:21

github-actions bot added the tests label Feb 24, 2026

nizamial09 force-pushed the isolate-cherrypy branch from 0b70cae to 13bc335 Compare February 24, 2026 06:23

nizamial09 mentioned this pull request Feb 24, 2026

debug #74980: prom_test_urls #67461

Closed

14 tasks

nizamial09 force-pushed the isolate-cherrypy branch from 13bc335 to dbe078a Compare February 24, 2026 13:40

rkachach approved these changes Feb 24, 2026

View reviewed changes

nizamial09 force-pushed the isolate-cherrypy branch from dbe078a to 1384ae3 Compare February 26, 2026 03:10

yaarith mentioned this pull request Feb 26, 2026

tentacle: mgr: isolated CherryPy to prevent global state sharing #67465

Open

nizamial09 added the ready-to-merge label Mar 2, 2026

nizamial09 merged commit 805aa0b into ceph:main Mar 3, 2026
13 of 15 checks passed

nizamial09 deleted the isolate-cherrypy branch March 3, 2026 07:14

github-project-automation bot moved this from Reviewer approved to Done in Ceph-Dashboard Mar 3, 2026

Conversation

nizamial09 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

nizamial09 commented Feb 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

NitzanMordhai commented Feb 5, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

athanatos left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

athanatos commented Feb 5, 2026

Uh oh!

nizamial09 commented Feb 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

nizamial09 commented Feb 11, 2026

Uh oh!

nizamial09 commented Feb 12, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rkachach left a comment

Choose a reason for hiding this comment

Uh oh!

ljflores commented Feb 26, 2026

Uh oh!

nizamial09 commented Feb 27, 2026

Uh oh!

Uh oh!

ljflores commented Mar 3, 2026

Uh oh!

Reviewers

Assignees

nizamial09 commented Feb 5, 2026 •

edited

Loading

nizamial09 commented Feb 5, 2026 •

edited

Loading