mgr: add config to load modules in main interpreter instead of subinterpreter#66244
mgr: add config to load modules in main interpreter instead of subinterpreter#66244
Conversation
Config Diff Tool Output+ added: mgr_main_interpreter_modules (mgr.yaml.in)
The above configuration changes are found in the PR. Please update the relevant release documentation if necessary. |
|
Replaces #66176 |
|
/config check ok |
|
depends on #66240 |
|
https://tracker.ceph.com/issues/73857 suggests that rbd_support needs to be in the main interpreter as well due to librbd/librados python bindings misbehaving. Perhaps the right choice would be to default modules into the main interpreter and make the config selectively move them into a subinterpreter? @cbodley @batrick @NitzanMordhai ? |
|
I have just asked this in https://tracker.ceph.com/issues/73857, but may get more eyes here: what was the actual problem that sub-interpreters were supposed to solve? |
d7b7361 to
2b3e1be
Compare
|
@idryomov That really is not clear to me. I think the idea was to provide isolation, but I don't think they ever actually did to any real degree. At this point, I think the question is moot until we have a reason to put effort into reenabling them (such as letting subinterpreters have distinct gils). |
6531997 to
a39c387
Compare
|
Marking back to draft for now. I see a potential problem with how MgrModule works -- MgrModule.COMMANDS seems like it'll end up with the command set for all modules in the same interpreter. I need to look into this more on Monday, might just be misunderstanding how it works. |
|
@athanatos i tried that code, it will load all modules into the main interpreter, which causes slow responses from any cli commands |
|
@athanatos I'm adding here example: you can see that when mgr mgr_subinterpreter_modules = "" (the default) it took 17.6 seconds (it can happened for any command) |
|
This PR is under test in https://tracker.ceph.com/issues/73884. |
|
@NitzanMordhai Good observation. I think it's related to the MgrModule.COMMANDS behavior I mentioned above -- I think the mgr is querying every module for every command. Will investigate. |
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
It was impossible to have multiple mgr modules using cherrypy at the same time. Here's the original email thread on this topic from back in April 2017: |
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
* refs/pull/66244/head: mgr/Gil.cc: simplify Gil(), ~Gil() mgr/Gil.cc: do not use PyGILState_Check() mgr: add mgr_subinterpreter_modules config python-common/.../service_spec: implement ServiceSpec.__getnewargs__ to allow unpickle to work correctly mgr: serialize python objects sent between subinterpreters via remote
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
After PR ceph#66244, all mgr modules run inside the same Python interpreter. That means they also share the same logging subsystem. Previously, each module attached its handlers to the root logger. In practice, whichever module initialized logging last effectively “owned” the root logger, and log messages from other modules could end up attributed incorrectly. This change scopes logging per module. Each module now registers its handlers on a dedicated logger named after the module itself, with propagate=False to avoid leaking messages into the root logger. Now, the getLogger() default (no args) returns the module's named logger rather than the root logger. This ensures self.log routes correctly. Fixes: https://tracker.ceph.com/issues/74848 Signed-off-by: Nitzan Mordechai <nmordec@ibm.com>
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com> (cherry picked from commit c0f77e4) Conflicts: src/pybind/mgr/cephadm/agent.py - removed the log filters and kept the ssl changes as it is src/pybind/mgr/dashboard/module.py - only added the new CherryPyMgr import src/pybind/mgr/prometheus/module.py - accepted the incoming changes
as the modules are now being loaded onto the main interpreter (see ceph/ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com> (cherry picked from commit c0f77e4) Conflicts: src/pybind/mgr/cephadm/agent.py - removed the log filters and kept the ssl changes as it is src/pybind/mgr/dashboard/module.py - only added the new CherryPyMgr import src/pybind/mgr/prometheus/module.py - accepted the incoming changes
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543, https://tracker.ceph.com/issues/74980 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543, https://tracker.ceph.com/issues/74980 Signed-off-by: Nizamudeen A <nia@redhat.com>
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543, https://tracker.ceph.com/issues/74980 Signed-off-by: Nizamudeen A <nia@redhat.com> (cherry picked from commit 13bc335)
as the modules are now being loaded onto the main interpreter (see ceph#66244), the cherrypy is getting hit with an issue where its global state is being affecting all the modules updating the cherrypy config simultaneously in the same tree. So i am adding a CherryPyMgr which manages all the independent servers that will be created across all modules. This CherryPyMgr will create its own server instances by utilizing cherrypy's WSGI Server and eliminates the global state sharing. Each module or app can create their own tree and start an adapter which will open an independent server for that app. - also added a method to update the config in place so CORS urls can be configured without restarting servers. Fixes: https://tracker.ceph.com/issues/74643, https://tracker.ceph.com/issues/74543, https://tracker.ceph.com/issues/74980 Signed-off-by: Nizamudeen A <nia@redhat.com>
|
This PR is under test in https://tracker.ceph.com/issues/75377. |
Show available Jenkins commands
jenkins test classic perfJenkins Job | Jenkins Job Definitionjenkins test crimson perfJenkins Job | Jenkins Job Definitionjenkins test signedJenkins Job | Jenkins Job Definitionjenkins test make checkJenkins Job | Jenkins Job Definitionjenkins test make check arm64Jenkins Job | Jenkins Job Definitionjenkins test submodulesJenkins Job | Jenkins Job Definitionjenkins test dashboardJenkins Job | Jenkins Job Definitionjenkins test dashboard cephadmJenkins Job | Jenkins Job Definitionjenkins test apiJenkins Job | Jenkins Job Definitionjenkins test docsReadTheDocs | Github Workflow Definitionjenkins test ceph-volume allJenkins Jobs | Jenkins Jobs Definitionjenkins test windowsJenkins Job | Jenkins Job Definitionjenkins test rook e2eJenkins Job | Jenkins Job DefinitionYou must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.