Ceph » mgr

I can confirm both aspects of this on our 17.2.5 cluster, with the RESTful module enabled: with the '[mon] allow profile mgr' caps Zabbix was unable to access many items, with '[mon] allow *' Zabbix access issues were resolved; we're also seeing a memory leak with this configuration, of a similar scale - something on the order of 20GB per day.

We were seeing growth up to ~70-80GB after five to seven days, at which point the RESTful api calls started failing or timing out. The mgr processes weren't being killed or failing over on their own, though (these nodes have 512GB RAM, so they weren't anywhere near an OOM situation), and as far as I can tell nothing else was impacted. Manually failing the active mgr seems to free up the memory without restarting the actual mgr process - we're working around the leak at the moment by failing the active mgr every 24 hours, which is clunky but hasn't been causing any issues for us.

Current versions are:

{
    "mon": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 5
    },
    "mgr": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 4
    },
    "osd": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 96
    },
    "mds": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 3
    },
    "rgw": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 4
    },
    "overall": {
        "ceph version 17.2.5 (98318ae89f1a893a6ded3a640405cdbb33e08757) quincy (stable)": 112
    }
}

Manager module list:

balancer              on (always on)
crash                 on (always on)
devicehealth          on (always on)
orchestrator          on (always on)
pg_autoscaler         on (always on)
progress              on (always on)
rbd_support           on (always on)
status                on (always on)
telemetry             on (always on)
volumes               on (always on)
cephadm               on            
dashboard             on            
iostat                on            
restful               on            
alerts                -             
diskprediction_local  -             
influx                -             
insights              -             
k8sevents             -             
localpool             -             
mds_autoscaler        -             
mirroring             -             
nfs                   -             
osd_perf_query        -             
osd_support           -             
prometheus            -             
rook                  -             
selftest              -             
snap_schedule         -             
stats                 -             
telegraf              -             
test_orchestrator     -             
zabbix                -

https://www.spinics.net/lists/ceph-users/msg76950.html

The mgr RESTful module is keeping all the finished requests, in some cases (like pg dump with hundreds of pgs) the output of the request that we are keeping can grow and grow without any limits.
i'm still trying to find a reason why we are keeping the finished results, if anyone has any good reason to keep them after some time please let me know.

I don't know if the RESTful module is still used. Dashboard has its own REST API inside the dashboard module which would probably eliminate the need for another RESTful module. I remember some discussion about deprecating or removing the RESTful module. Not sure if its still in plan to deprecate it.

Nizamudeen A thanks for the input, what about zabbix using that restful api?

The problem is that we are keeping the results of each request, when the result is small, that's fine, but when it comes to pg dumps and other commands that potentially can return a lot of data, the result array can grow and finally cause OOM

    def finish(self, tag):
        with self.lock:
            for index in range(len(self.running)):
                if self.running[index].tag == tag:
                    if self.running[index].r == 0:
                        self.finished.append(self.running.pop(index))
                    else:
                        self.failed.append(self.running.pop(index))
                    return True

            # No such tag found
            return False

There is no limit on self.failed or self.finished.

Hi all,

we are experiencing memory leak in MGR as well. However we have restful module turned off and CAPS are set to allow profile mgr. Strange thing is that at one point memory starts growing on multiple Ceph clusters within one datacenter rapidly.

Version 17.2.6

Any chance of progressing this?

Regardless of whether there is actually any reason for keeping the results (??), continually adding to a memory structure without bounds is not a great idea!

Thanks, Chris

We're seeing this anywhere from 1-3 times a day to once a week on all clusters running .13 and .14. We externally poll the metrics endpoint frequently, in case it's related. We see this on clusters we do not utilize the RESTFul module with.

Greg Farnum wrote:

There are two separate reports on the mailing list of memory leaks in the mgr module:

[ceph-users] Memory leak in MGR after upgrading to pacific
[...]

[ceph-users] MGR Memory Leak in Restful
[...]

It happend us three times in last month.

Likely it changed something in between 16.2.10-16.2.13 ... that causes this to happen. We have two reports that after update from .10 to .13 ceph mgr got oom killed several times.

[root@ctplmon1 bin]# ceph mgr services
{
    "prometheus": "http://xxx:9119/" 
}

{
    "always_on_modules": [
        "balancer",
        "crash",
        "devicehealth",
        "orchestrator",
        "pg_autoscaler",
        "progress",
        "rbd_support",
        "status",
        "telemetry",
        "volumes" 
    ],
    "enabled_modules": [
        "iostat",
        "prometheus",
        "restful" 
    ] ...
}

David Orman wrote:

We're seeing this anywhere from 1-3 times a day to once a week on all clusters running .13 and .14. We externally poll the metrics endpoint frequently, in case it's related. We see this on clusters we do not utilize the RESTFul module with.

That sounds like a different issue if the restful module was not utilized on those mgrs, can you run the mgr with massif profiling and send us the output?

I tried applying this patch to 17.2.6. After applying the CAPS change and restarting the active MGR these type of entries were logged in the MGR log:

2023-09-21T14:21:21.272+0100 7f22f3fba700  0 [restful ERROR root] Traceback (most recent call last):
  File "/lib/python3/dist-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/lib/python3/dist-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/decorators.py", line 37, in decorated
    return f(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/api/request.py", line 88, in post
    return context.instance.submit_request([[request.json]], **kwargs)
  File "/usr/share/ceph/mgr/restful/module.py", line 605, in submit_request
    request = CommandsRequest(_request)
  File "/usr/share/ceph/mgr/restful/module.py", line 62, in __init__
    max_finished = cast(int, self.get_localized_module_option('max_finished', 100))
AttributeError: 'CommandsRequest' object has no attribute 'get_localized_module_option'

2023-09-21T14:21:21.280+0100 7f22f3fba700  0 [restful ERROR werkzeug] Error on request:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 323, in run_wsgi
    execute(self.server.app)
  File "/usr/lib/python3/dist-packages/werkzeug/serving.py", line 312, in execute
    application_iter = app(environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/middleware/recursive.py", line 56, in __call__
    return self.application(environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 840, in __call__
    return super(Pecan, self).__call__(environ, start_response)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/usr/lib/python3/dist-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/decorators.py", line 37, in decorated
    return f(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/api/request.py", line 88, in post
    return context.instance.submit_request([[request.json]], **kwargs)
  File "/usr/share/ceph/mgr/restful/module.py", line 605, in submit_request
    request = CommandsRequest(_request)
  File "/usr/share/ceph/mgr/restful/module.py", line 62, in __init__
    max_finished = cast(int, self.get_localized_module_option('max_finished', 100))
AttributeError: 'CommandsRequest' object has no attribute 'get_localized_module_option'

2023-09-21T14:25:24.095+0100 7f22f3fba700  0 [restful ERROR root] Traceback (most recent call last):
  File "/lib/python3/dist-packages/pecan/core.py", line 683, in __call__
    self.invoke_controller(controller, args, kwargs, state)
  File "/lib/python3/dist-packages/pecan/core.py", line 574, in invoke_controller
    result = controller(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/decorators.py", line 37, in decorated
    return f(*args, **kwargs)
  File "/usr/share/ceph/mgr/restful/api/request.py", line 88, in post
    return context.instance.submit_request([[request.json]], **kwargs)
  File "/usr/share/ceph/mgr/restful/module.py", line 605, in submit_request
    return request
  File "/usr/share/ceph/mgr/restful/module.py", line 62, in __init__
    self.lock = threading.RLock()
AttributeError: 'CommandsRequest' object has no attribute 'get_localized_module_option'

Thanks for continuing to look at this. Has this patch been tested against Quincy (17.2.6)? It still has a reference to get_localized_module_option which will probably cause the same errors as above...

Chris Palmer wrote:

Thanks for continuing to look at this. Has this patch been tested against Quincy (17.2.6)? It still has a reference to get_localized_module_option which will probably cause the same errors as above...

yes, it still has get_localized_module_option, but now it's not under the server, it's under the init of the module itself.

Doesn't look very promising I'm afraid. I just tried the patch on a small 17.2.6 cluster with 745 pgs and zabbix monitoring enabled and "mon 'allow *'" in the active manager caps. As before the monitoring worked. However the mgr rss started at 393M and 12 hours later was at 2.2G and growing, so around 4GB per 24 hours. This is consistent with the original report where a cluster with about 5 times the number of pgs leaked around 20GB per 24 hours.

I just had a 17.2.6 MGR on another cluster go OOM. It was on a bigger cluster, with Zabbix integration, but without the 'allow *' CAPS so most of the calls were getting denied. This MGR had been the active one for some months. It all suggests a leak in the REST interface but it is probably not restricted to the pgs dump function.

Chris Palmer wrote:

I just had a 17.2.6 MGR on another cluster go OOM. It was on a bigger cluster, with Zabbix integration, but without the 'allow *' CAPS so most of the calls were getting denied. This MGR had been the active one for some months. It all suggests a leak in the REST interface but it is probably not restricted to the pgs dump function.

Chris, i'm running tests on my environment and I can see the leaks from the restful module without the patch, with the patch the memory doesn't increass.
i'm starting to wonder if we have also Zabbix module memory leak, can you try the patch after you disabled Zabbix module?

Ah, slight confusion. We are not using the Ceph Zabbix module. That module runs inside the manager and pushes data to Zabbix, but is not very comprehensive. Instead we use a more recent Zabbix template that runs inside a Zabbix agent on another host, and simply polls the active Ceph MGR using the Rest API. So it is entirely consistent that the leak is in the Rest API itself. Zabbix is simply calling it more often that it might normally be. When I enable that additional CAP it performs a lot more calls which turns a small problem into a larger one.

I suspect that your patch fixes a secondary memory buildup, but that the primary leak hasn't been found yet.

Chris Palmer wrote:

Ah, slight confusion. We are not using the Ceph Zabbix module. That module runs inside the manager and pushes data to Zabbix, but is not very comprehensive. Instead we use a more recent Zabbix template that runs inside a Zabbix agent on another host, and simply polls the active Ceph MGR using the Rest API. So it is entirely consistent that the leak is in the Rest API itself. Zabbix is simply calling it more often that it might normally be. When I enable that additional CAP it performs a lot more calls which turns a small problem into a larger one.

I suspect that your patch fixes a secondary memory buildup, but that the primary leak hasn't been found yet.

Chris, i had some attempts to recreate the issue, with the patch, but i still couldn't get it to increase quickly as you are seeing.
can you run the mgr with valgrind massif and provide us the output?

I'll give it a go. I have never used massif before though. Do you have any instructions for how to run the mgr up under it, including any command line options I should use? (It is running non-containerized under debian11).

I'm using:

valgrind --tool=massif bin/ceph-mgr -f --cluster ceph -i x &

valgrind massif is pretty basic, non argument is needed.

I'm examining your massif file and trying to come with a fix. i still was not able to cause the memory to leak quick enough. any chance you are using mgr_ttl_cache_expire_seconds ?

Thanks. There are no mgr-related tuning parameters set, and specifically mgr_ttl_cache_expire_seconds has not been set. Let me know if I can do anything else.

Any news about this issue?
We have 2 clusters (16.2.14) heavily affected by this memory leak.
Is there a way to limit memory usage for MGR?

Same thing happened with the restful module disabled

    "enabled_modules": [
        "cephadm",
        "dashboard",
        "iostat",
        "nfs",
        "prometheus" 
    ]

Adrien Georget wrote:

Any news about this issue?
We have 2 clusters (16.2.14) heavily affected by this memory leak.
Is there a way to limit memory usage for MGR?

Same thing happened with the restful module disabled
[...]

I'm still working on that issue, i couldn't make the mgr to leak memory with the above rates.
can you also send massif output with disabled restful module in mgr?

Adrien Georget wrote:

Is there a way to limit memory usage for MGR?

As a temporary solution, I have limited the memory usage in the systemd unit. The MGR will be killed before it wastes all the memory and compromises the entire node.

Adrien, can you please add a massif output file from the mgr (one without restful enable) that leaking memory?

Our 16.2.14 cluster is affected as well. Modules enabled:

"always_on_modules": [
        "balancer",
        "crash",
        "devicehealth",
        "orchestrator",
        "pg_autoscaler",
        "progress",
        "rbd_support",
        "status",
        "telemetry",
        "volumes" 
    ],
    "enabled_modules": [
        "cephadm",
        "dashboard",
        "iostat",
        "prometheus",
        "restful" 
    ],

No special mgr settings, all defaults.

Hello folks

After upgrading to version 16.2.13, we encountered MGR OOM issues. It's worth noting that the restful module was not enabled at the time.
The MGR would experience OOM issues about once a week. The memory usage would start increasing without any apparent reason and in less than an hour, it would take up all 300GiB.
However, when we upgraded to Quincy, the issue disappeared.

Hi Team,

We are affected as well. Since we upgraded to 16.2.14 on two diff cluster 'ceph-mgr' oom and gets killed. It's random. Happens 3 to 5 times per month. Did not have this issue on 16.2.11.

waitting for https://github.com/ceph/ceph/pull/54984 merge and backport

Hi,
Same observation, 16.2.15 did not fix this issue.
ceph-mgr still crash with OOM.

Jul 19 03:46:14 ceph-mgr[2711105]: tcmalloc: large alloc 1233903616 bytes == 0x55c9a211c000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:46:31 ceph-mgr[2711105]: tcmalloc: large alloc 1542381568 bytes == 0x55c933098000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:46:40 ceph-mgr[2711105]: tcmalloc: large alloc 1927979008 bytes == 0x55c9eb9da000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:47:04 ceph-mgr[2711105]: tcmalloc: large alloc 2409979904 bytes == 0x55ca5e884000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:47:44 ceph-mgr[2711105]: tcmalloc: large alloc 3012476928 bytes == 0x55c9a211c000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:50:41 ceph-mgr[2711105]: tcmalloc: large alloc 3765600256 bytes == 0x55caee2da000 @  0x7f5d4c8df760 0x7f5d4c900a62 0x7f5d55ea7988 0x7f5d55ed74d5 0x55c8d30128bb 0x55c8d3012ba0 0x55c8d2f33d9d 0x7f5d55f55057 0x7f5d55f55f48 0x7f5d55f32f08 0x>
Jul 19 03:51:02 systemd[1]: ceph-mgr@adm20.service: Main process exited, code=killed, status=9/KILL
Jul 19 03:51:02 systemd[1]: ceph-mgr@adm20.service: Failed with result 'signal'.
Jul 19 03:51:12 systemd[1]: ceph-mgr@adm20.service: Service RestartSec=10s expired, scheduling restart.
Jul 19 03:51:12 systemd[1]: ceph-mgr@adm20.service: Scheduled restart job, restart counter is at 1.
Jul 19 03:51:12 systemd[1]: Stopped Ceph cluster manager daemon.
Jul 19 03:51:12 systemd[1]: Started Ceph cluster manager daemon.

Hi,

Is there a workaround? It happens once or twice every month on 16.2.14 and 16.2.15 .

@rsachere@redhat.com i need to backport https://github.com/ceph/ceph/pull/54984 when it will be testes.

@Laura Flores can we add it to one of the tests soon?

Nitzan Mordechai wrote in #note-57:

@rsachere@redhat.com i need to backport https://github.com/ceph/ceph/pull/54984 when it will be testes.

@Laura Flores can we add it to one of the tests soon?

@Nitzan Mordechai sure! I added the core label to it- that part was missing to make sure it gets placed in one of Yuri's batches for core.

We were operating a cluster with our own build based on version 16.2.10 and did not experience any OOM issues with ceph-mgr.
However, after upgrading to a version built on 16.2.15, we began to encounter OOM problems. For reference, the Prometheus module is enabled.
Typically, these issues occurred within two weeks after starting ceph-mgr.
To investigate, we conducted profiling using heaptrack and discovered that there was a memory leak in get_daemon_health_metrics(https://github.com/ceph/ceph/pull/48843).
We reverted the commits related to get_daemon_health_metrics and rebuilt the version.
It has now been 45 days since we started running ceph-mgr, and it remains stable.

Beom-Seok Park wrote in #note-60:

We were operating a cluster with our own build based on version 16.2.10 and did not experience any OOM issues with ceph-mgr.
However, after upgrading to a version built on 16.2.15, we began to encounter OOM problems. For reference, the Prometheus module is enabled.
Typically, these issues occurred within two weeks after starting ceph-mgr.
To investigate, we conducted profiling using heaptrack and discovered that there was a memory leak in get_daemon_health_metrics(https://github.com/ceph/ceph/pull/48843).
We reverted the commits related to get_daemon_health_metrics and rebuilt the version.
It has now been 45 days since we started running ceph-mgr, and it remains stable.

Can you create dedicated issue for your investigation please? Seems this exactly like our reports

Konstantin Shalygin wrote in #note-61:

Can you create dedicated issue for your investigation please? Seems this exactly like our reports

I have created a new issue.
https://tracker.ceph.com/issues/68989