Project

General

Profile

Actions
Copy link

Bug #74604

open

Rocky10 - MismatchedFree delete coming from ceph-osd-classic code

Added by Laura Flores about 2 months ago. Updated 4 days ago.

Status:
Fix Under Review
Priority:
Normal
Assignee:
Nitzan Mordechai
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
67230
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Tags:
rocky10

Description

/a/nmordech-2026-01-25_11:10:14-rados-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/17041

<error>
  <unique>0x21c</unique>
  <tid>1</tid>
  <kind>MismatchedFree</kind>
  <what>Mismatched free() / delete / delete []</what>
  <stack>
    <frame>
      <ip>0x61275EC</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete[](void*, unsigned long, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>1504</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>new_allocator.h</file>
      <line>167</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>allocator.h</file>
      <line>208</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>alloc_traits.h</file>
      <line>550</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>563</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>630</line>
    </frame>
    <frame>
      <ip>0x4B989D6</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>std::_Rb_tree&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::type_index&gt;, std::pair&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::type_index&gt; const, ceph::immobile_any&lt;576ul&gt; &gt;, std::_Select1st&lt;std::pair&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::type_index&gt; const, ceph::immobile_any&lt;576ul&gt; &gt; &gt;, ceph::common::CephContext::associated_objs_cmp, std::allocator&lt;std::pair&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::type_index&gt; const, ceph::immobile_any&lt;576ul&gt; &gt; &gt; &gt;::_M_erase_aux(std::_Rb_tree_const_iterator&lt;std::pair&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::type_index&gt; const, ceph::immobile_any&lt;576ul&gt; &gt; &gt;)</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>2494</line>
    </frame>
    <frame>
      <ip>0x4B3DBBF</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>global_init_prefork(ceph::common::CephContext*)</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/global</dir>
      <file>global_init.cc</file>
      <line>506</line>
    </frame>
    <frame>
      <ip>0x44D23A5</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>main</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src</dir>
      <file>ceph_osd.cc</file>
      <line>210</line>
    </frame>
  </stack>
  <auxwhat>Address 0x8329800 is 0 bytes inside a block of size 4,096 alloc'd</auxwhat>
  <stack>
    <frame>
      <ip>0x6121732</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>547</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>new_allocator.h</file>
      <line>147</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>allocator.h</file>
      <line>196</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>alloc_traits.h</file>
      <line>515</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>559</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>609</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>1633</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_tree.h</file>
      <line>2460</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/include/c++/14/bits</dir>
      <file>stl_map.h</file>
      <line>640</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>UnknownInlinedFun</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/common</dir>
      <file>ceph_context.h</file>
      <line>227</line>
    </frame>
    <frame>
      <ip>0x4D11B09</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>AsyncMessenger::AsyncMessenger(ceph::common::CephContext*, entity_name_t, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/msg/async</dir>
      <file>AsyncMessenger.cc</file>
      <line>379</line>
    </frame>
    <frame>
      <ip>0x4D12240</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>Messenger::create(ceph::common::CephContext*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const&amp;, entity_name_t, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, unsigned long)</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/msg</dir>
      <file>Messenger.cc</file>
      <line>36</line>
    </frame>
    <frame>
      <ip>0x4D12422</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>Messenger::create_client_messenger(ceph::common::CephContext*, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;)</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/msg</dir>
      <file>Messenger.cc</file>
      <line>17</line>
    </frame>
    <frame>
      <ip>0x4D610A2</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>MonClient::get_monmap_and_config()</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/mon</dir>
      <file>MonClient.cc</file>
      <line>133</line>
    </frame>
    <frame>
      <ip>0x4B4314F</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>global_init(std::map&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt;, std::less&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt;, std::allocator&lt;std::pair&lt;std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; const, std::__cxx11::basic_string&lt;char, std::char_traits&lt;char&gt;, std::allocator&lt;char&gt; &gt; &gt; &gt; &gt; const*, std::vector&lt;char const*, std::allocator&lt;char const*&gt; &gt;&amp;, unsigned int, code_environment_t, int, bool)</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src/global</dir>
      <file>global_init.cc</file>
      <line>409</line>
    </frame>
    <frame>
      <ip>0x44D20CE</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>main</fn>
      <dir>/usr/src/debug/ceph-20.3.0-4942.gb62a951f.el10.x86_64/src</dir>
      <file>ceph_osd.cc</file>
      <line>140</line>
    </frame>
  </stack>

Related issues 3 (3 open — 0 closed)

Related to RADOS - Bug #73822: Rocky10 - rados/verify - valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) RocksDBStore::close() RocksDBStore::~RocksDBStore() Fix Under Review

Actions
Related to Ceph QA - QA Run #74540: wip-rocky10-branch-of-the-day-2026-01-23-1769128778QA Needs ApprovalLaura FloresActions
Related to Ceph QA - QA Run #75339: wip-rocky10-branch-of-the-day-2026-03-04-1772633736QA TestingYaarit HatukaActions
Actions
Copy link
 #1

Updated by Laura Flores about 2 months ago

  • Description updated (diff)
Actions
Copy link
 #2

Updated by Laura Flores about 2 months ago

  • Related to Bug #73822: Rocky10 - rados/verify - valgrind error: MismatchedFree operator delete[](void*, unsigned long, std::align_val_t) RocksDBStore::close() RocksDBStore::~RocksDBStore() added
Actions
Copy link
 #3

Updated by Laura Flores about 2 months ago

  • Related to QA Run #74540: wip-rocky10-branch-of-the-day-2026-01-23-1769128778 added
Actions
Copy link
 #4

Updated by Laura Flores about 2 months ago

Bump up - part of the Rocky10 work

Actions
Copy link
 #5

Updated by Nitzan Mordechai about 2 months ago

  • Status changed from New to In Progress
  • Assignee set to Nitzan Mordechai
Actions
Copy link
 #6

Updated by Nitzan Mordechai about 1 month ago

/a/nmordech-2026-02-04_08:13:49-rados-wip-rocky10-branch-of-the-day-2026-02-03-1770151121-distro-default-trial/33913/
/a/nmordech-2026-02-04_08:13:49-rados-wip-rocky10-branch-of-the-day-2026-02-03-1770151121-distro-default-trial/33990/

Actions
Copy link
 #7

Updated by Nitzan Mordechai about 1 month ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 67230
Actions
Copy link
 #8

Updated by Radoslaw Zarzynski about 1 month ago

Sent to QA.

Actions
Copy link
 #9

Updated by Nitzan Mordechai about 1 month ago

  • Backport set to tentacle
Actions
Copy link
 #10

Updated by Nitzan Mordechai about 1 month ago

/a/yaarit-2026-02-10_02:34:33-rados-wip-rocky10-branch-of-the-day-2026-02-09-1770676549-distro-default-trial/42172

Actions
Copy link
 #11

Updated by Nitzan Mordechai about 1 month ago ยท Edited

/a/yaarit-2026-02-10_23:48:52-rados-wip-rocky10-branch-of-the-day-2026-02-09-1770676549-distro-default-trial/
['44505', '44285']

need to remove the dealloc

Actions
Copy link
 #12

Updated by Radoslaw Zarzynski about 1 month ago

In QA.

Actions
Copy link
 #13

Updated by Laura Flores 25 days ago

Testing through inclusion in the "rocky10 of the day" branch.

Actions
Copy link
 #14

Updated by Radoslaw Zarzynski 18 days ago

In QA.

Actions
Copy link
 #15

Updated by Laura Flores 15 days ago

/a/yaarit-2026-03-05_02:43:32-rados-wip-rocky10-branch-of-the-day-2026-03-04-1772633736-distro-default-trial/86266/remote/trial015/log/valgrind/osd.0.log.gz

<error>
  <unique>0x1bf98</unique>
  <tid>1</tid>
  <kind>MismatchedFree</kind>
  <what>Mismatched free() / delete / delete []</what>
  <stack>
    <frame>
      <ip>0x61350CE</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete[](void*, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>1491</line>
    </frame>
    <frame>
      <ip>0x6DA65B0</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__run_exit_handlers</fn>
    </frame>
    <frame>
      <ip>0x6DA666F</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>exit</fn>
    </frame>
    <frame>
      <ip>0x6D8D594</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>(below main)</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x7674700 is 0 bytes inside a block of size 16,384 alloc'd</auxwhat>
  <stack>
    <frame>
      <ip>0x612F732</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>547</line>
    </frame>
    <frame>
      <ip>0x452A487</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>_sub_I_65535_1.1</fn>
    </frame>
    <frame>
      <ip>0x6D8D6BD</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__libc_start_main@@GLIBC_2.34</fn>
    </frame>
    <frame>
      <ip>0x452BA24</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>(below main)</fn>
    </frame>
  </stack>
  <suppression>
    <sname>insert_a_suppression_name_here</sname>
    <skind>Memcheck:Free</skind>
    <sframe> <fun>_ZdaPvSt11align_val_t</fun> </sframe>
    <sframe> <fun>__run_exit_handlers</fun> </sframe>
    <sframe> <fun>exit</fun> </sframe>
    <sframe> <fun>(below main)</fun> </sframe>
    <rawtext>
<![CDATA[
{
   <insert_a_suppression_name_here>
   Memcheck:Free
   fun:_ZdaPvSt11align_val_t
   fun:__run_exit_handlers
   fun:exit
   fun:(below main)
}
]]>
    </rawtext>
  </suppression>
</error>

  <suppression>
    <sname>insert_a_suppression_name_here</sname>
    <skind>Memcheck:Free</skind>
    <sframe> <fun>_ZdaPvSt11align_val_t</fun> </sframe>
    <sframe> <fun>__run_exit_handlers</fun> </sframe>
    <sframe> <fun>exit</fun> </sframe>
    <sframe> <fun>(below main)</fun> </sframe>
    <rawtext>
<![CDATA[
{
   <insert_a_suppression_name_here>
   Memcheck:Free
   fun:_ZdaPvSt11align_val_t
   fun:__run_exit_handlers
   fun:exit
   fun:(below main)
}
]]>

Actions
Copy link
 #16

Updated by Laura Flores 15 days ago

  • Related to QA Run #75339: wip-rocky10-branch-of-the-day-2026-03-04-1772633736 added
Actions
Copy link
 #17

Updated by Radoslaw Zarzynski 11 days ago

Approved the PR.

Actions
Copy link
 #18

Updated by Radoslaw Zarzynski 4 days ago

In QA.

Actions
Copy link

Also available in: Atom PDF