Add memory usage logging during the tick for both OSD and MON components by shreekarSS · Pull Request #54698 · ceph/ceph

shreekarSS · 2023-11-28T19:44:56Z

Description

This pull request adds memory usage logging during the tick operation for OSD and MON components. The aim is to improve visibility into memory consumption trends, facilitating monitoring and debugging.

Changes

Implemented memory usage logging with the MemoryModel class.
Established a baseline for reference.
Included detailed memory metrics (total, rss, heap) in the tick log output.

Context

This enhancement addresses the need for better memory insights in OSD and MON components, enhancing overall system health and performance monitoring.

Related Issue

Fixes: https://tracker.ceph.com/issues/54525

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

shreekarSS · 2023-12-01T18:37:46Z

From OSD log:
2023-11-29T05:03:42.283+0000 ffff7cbec140 2 osd.0 56 OSD Memory usage: total 748500, rss 194400, heap 518416, baseline 518416

From MON log:
2023-11-29T14:29:08.984+0000 ffff9394bb00 2 mon.a@0(leader) e1 MON Memory usage: total 718084, rss 417584, heap 199104, baseline 190912

src/mon/Monitor.cc

src/mon/CMakeLists.txt

athanatos · 2024-01-04T02:18:29Z

src/mon/Monitor.cc

+  static MemoryModel mm(g_ceph_context);
+  static MemoryModel::snap last;
+  mm.sample(&last);
+  static MemoryModel::snap baseline = last;


This does appear to identical to the existing code in MDCache, but there are a few things I don't understand:

Won't last and baseline always match?

Why do mm, last, and baseline need to be static? I'm not really ok with making these static without a specific reason.

It seems like mm and last can simply be local variables. I presume that the intent is for baseline to contain the last recorded value so that the log line can contain the last value and the current value, so baseline should be a member of Monitor.

hi @athanatos,
I have implemented the recommended changes. The 'baseline' variable is now set as a private member in the Monitor, OSD, and MDS classes. This enhancement streamlines our approach to memory tracking and analysis. Thank you for your constructive feedback.

…sion in ceph#54698 (comment). - 'baseline' is now a member of the Monitor class for enhanced tracking and logging. Signed-off-by: ShreekarSS <Shreekara.ss@gmail.com>

athanatos · 2024-01-31T23:22:15Z

@ceph/cephfs There's a minor cleanup to the MDCache in this PR -- can you take a look? @Ceph/core I ended up rewriting several of the commits here, so I probably shouldn't review it.

src/mds/MDCache.cc

dparmar18

MDCache code changes LGTM.

vshankar

LGTM from cephfs POV.

github-actions · 2024-08-05T12:02:01Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

yuriw · 2024-08-08T14:52:28Z

@shreekarSS pls rebase

We're going to add this into the OSD and the Mon. Signed-off-by: Samuel Just <sjust@redhat.com>

Fixes: https://tracker.ceph.com/issues/54525 Signed-off-by: Samuel Just <sjust@redhat.com> Signed-off-by: Shreekara <sshreeka@redhat.com>

batrick · 2024-09-14T16:31:34Z

src/mon/Monitor.h

  void new_tick();

+  // Memory usage baseline snapshot for monitoring
+  MemoryModel::snap baseline;


Please also fix MDCache to not have a static duration variable: #56812 (comment)

batrick · 2024-09-14T16:35:13Z

src/mon/Monitor.cc

+	  << " total " << mm.last.get_total()
+	  << ", rss " << mm.last.get_rss()
+	  << ", heap " << mm.last.get_heap()
+	  << ", baseline " << baseline.get_heap()


Please add void MemoryModel::mem_snap_t::print(std::ostream&) and make the printing consistent for all of these douts so you can:

dout(2) << "MON memory usage: " << mm.last << dendl;

and add units (i.e. kiB IIRC). The unit-less prints have been a source of confusion in the past for support engineers.

yuriw · 2024-10-01T15:02:05Z

jenkins test this please

ljflores · 2024-10-03T20:10:33Z

@shreekarSS ping

yuriw · 2025-01-17T16:02:30Z

@shreekarSS do we still want this?

ceph-jenkins · 2025-01-17T16:02:33Z

Can one of the admins verify this patch?

github-actions · 2025-03-18T17:01:33Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2025-04-16T13:02:24Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

github-actions · 2025-06-15T15:01:32Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

ronen-fr · 2025-06-16T13:28:11Z

@shreekarSS ping

github-actions · 2025-08-15T15:01:36Z

This pull request has been automatically marked as stale because it has not had any activity for 60 days. It will be closed if no further activity occurs for another 30 days.
If you are a maintainer or core committer, please follow-up on this pull request to identify what steps should be taken by the author to move this proposed change forward.
If you are the author of this pull request, thank you for your proposed contribution. If you believe this change is still appropriate, please ensure that any feedback has been addressed and ask for a code review.

github-actions · 2025-09-14T16:01:48Z

This pull request has been automatically closed because there has been no activity for 90 days. Please feel free to reopen this pull request (or open a new one) if the proposed change is still appropriate. Thank you for your contribution!

shreekarSS requested a review from a team as a code owner November 28, 2023 19:44

github-actions bot added build/ops core mon labels Nov 28, 2023

shreekarSS force-pushed the log-memory-during-tick-fix branch from 0ba6a23 to 92681bd Compare November 28, 2023 20:54

rzarzynski reviewed Dec 1, 2023

View reviewed changes

src/mon/Monitor.cc Outdated Show resolved Hide resolved

src/mon/CMakeLists.txt Outdated Show resolved Hide resolved

athanatos reviewed Jan 4, 2024

View reviewed changes

shreekarSS force-pushed the log-memory-during-tick-fix branch 2 times, most recently from 92681bd to 997f35d Compare January 10, 2024 05:52

shreekarSS force-pushed the log-memory-during-tick-fix branch from eecd2d9 to 850983a Compare January 12, 2024 20:52

github-actions bot added the cephfs Ceph File System label Jan 22, 2024

shreekarSS force-pushed the log-memory-during-tick-fix branch 5 times, most recently from a23ff89 to 6185b9f Compare January 26, 2024 23:24

github-actions bot added the common label Jan 26, 2024

shreekarSS force-pushed the log-memory-during-tick-fix branch from 6185b9f to e7a0d23 Compare January 27, 2024 06:38

athanatos requested review from a team and athanatos and removed request for athanatos January 31, 2024 23:19

dparmar18 reviewed Feb 1, 2024

View reviewed changes

src/mds/MDCache.cc Outdated Show resolved Hide resolved

dparmar18 reviewed Feb 1, 2024

View reviewed changes

src/mds/MDCache.cc Show resolved Hide resolved

dparmar18 reviewed Feb 1, 2024

View reviewed changes

vshankar approved these changes Feb 12, 2024

View reviewed changes

github-actions bot removed the needs-rebase label Jul 31, 2024

shreekarSS force-pushed the log-memory-during-tick-fix branch from b613075 to 3cc340e Compare July 31, 2024 21:52

github-actions bot added the needs-rebase label Aug 5, 2024

athanatos added 3 commits August 8, 2024 22:26

common/CMakeLists: add MemoryModel to libcommon

7d16f11

We're going to add this into the OSD and the Mon. Signed-off-by: Samuel Just <sjust@redhat.com>

mon: add logging and perfcounters for rss and heap usage

a17e13c

Fixes: https://tracker.ceph.com/issues/54525 Signed-off-by: Samuel Just <sjust@redhat.com> Signed-off-by: Shreekara <sshreeka@redhat.com>

osd: add logging and perfcounters for rss and heap usage

0e6d46d

Fixes: https://tracker.ceph.com/issues/54525 Signed-off-by: Samuel Just <sjust@redhat.com> Signed-off-by: Shreekara <sshreeka@redhat.com>

shreekarSS force-pushed the log-memory-during-tick-fix branch from 3cc340e to 0e6d46d Compare August 9, 2024 05:34

github-actions bot removed the needs-rebase label Aug 9, 2024

batrick requested changes Sep 14, 2024

View reviewed changes

yuriw removed the ready-to-merge label Oct 1, 2024

mchangir removed the wip-mchangir-testing not yet production ready label Nov 22, 2024

github-actions bot added the stale label Mar 18, 2025

github-actions bot added the needs-rebase label Apr 16, 2025

github-actions bot removed the stale label Apr 16, 2025

github-actions bot added the stale label Jun 15, 2025

github-actions bot removed the stale label Jun 16, 2025

github-actions bot added the stale label Aug 15, 2025

github-actions bot closed this Sep 14, 2025

Conversation

shreekarSS commented Nov 28, 2023 • edited by athanatos Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes

Context

Related Issue

Uh oh!

shreekarSS commented Dec 1, 2023

Uh oh!

Uh oh!

Uh oh!

athanatos Jan 4, 2024

Choose a reason for hiding this comment

Uh oh!

shreekarSS Jan 24, 2024

Choose a reason for hiding this comment

Uh oh!

athanatos commented Jan 31, 2024

Uh oh!

Uh oh!

Uh oh!

dparmar18 left a comment

Choose a reason for hiding this comment

Uh oh!

vshankar left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Aug 5, 2024

Uh oh!

yuriw commented Aug 8, 2024

Uh oh!

batrick Sep 14, 2024

Choose a reason for hiding this comment

Uh oh!

batrick Sep 14, 2024

Choose a reason for hiding this comment

Uh oh!

yuriw commented Oct 1, 2024

Uh oh!

ljflores commented Oct 3, 2024

Uh oh!

yuriw commented Jan 17, 2025

Uh oh!

ceph-jenkins commented Jan 17, 2025

Uh oh!

github-actions bot commented Mar 18, 2025

Uh oh!

github-actions bot commented Apr 16, 2025

Uh oh!

github-actions bot commented Jun 15, 2025

Uh oh!

ronen-fr commented Jun 16, 2025

Uh oh!

github-actions bot commented Aug 15, 2025

Uh oh!

github-actions bot commented Sep 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

12 participants

shreekarSS commented Nov 28, 2023 •

edited by athanatos

Loading