Skip to content

qa: suppress false positive delete map mismatch errors#67230

Open
NitzanMordhai wants to merge 1 commit intoceph:mainfrom
NitzanMordhai:wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14
Open

qa: suppress false positive delete map mismatch errors#67230
NitzanMordhai wants to merge 1 commit intoceph:mainfrom
NitzanMordhai:wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14

Conversation

@NitzanMordhai
Copy link
Contributor

@NitzanMordhai NitzanMordhai commented Feb 5, 2026 •
edited
Loading

Valgrind reports "Mismatched free() / delete / delete []" errors during
OSD startup.

Standard library containers (like std::map) correctly call delete, but
Valgrind falsely interprets this as a call to delete[] because GCC 14
folds the identical aligned delete operators into a single symbol. This
causes Valgrind to flag a mismatch against the non-array allocation.

Fixes: https://tracker.ceph.com/issues/74604

Contribution Guidelines

  • To sign and title your commits, please refer to Submitting Patches to Ceph.

  • If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.

  • When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

  • Tracker (select at least one)
    • References tracker ticket
    • Very recent bug; references commit where it was introduced
    • New feature (ticket optional)
    • Doc update (no ticket needed)
    • Code cleanup (no ticket needed)
  • Component impact
    • Affects Dashboard, opened tracker ticket
    • Affects Orchestrator, opened tracker ticket
    • No impact that needs to be tracked
  • Documentation (select at least one)
    • Updates relevant documentation
    • No doc update is appropriate
  • Tests (select at least one)
Show available Jenkins commands

You must only issue one Jenkins command per-comment. Jenkins does not understand
comments with more than one command.

@github-actions github-actions bot added the tests label Feb 5, 2026
@ljflores ljflores added the core label Feb 9, 2026
rzarzynski
rzarzynski approved these changes Feb 9, 2026
View reviewed changes
Copy link
Contributor

@rzarzynski rzarzynski left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM assuming it passes QA.

@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14 branch from 56de494 to c50f7e6 Compare February 11, 2026 11:56
batrick added a commit to batrick/ceph that referenced this pull request Feb 12, 2026
@batrick
Merge PR ceph#67230 into wip-pdonnell-testing-20260210.171210
cd41d55 
* refs/pull/67230/head:
	qa: suppress false positive delete map mismatch errors
	qa/workunits/ceph-helpers-root: Add Rocky support for intall packages
@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14 branch from c50f7e6 to 7fb83c0 Compare February 17, 2026 11:09
@ljflores
Copy link
Member

ljflores commented Feb 23, 2026

jenkins retest this please

@NitzanMordhai NitzanMordhai mentioned this pull request Feb 25, 2026
Draft
@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14 branch from 7fb83c0 to ac1b45b Compare March 4, 2026 12:55
@ljflores
Copy link
Member

ljflores commented Mar 5, 2026 •
edited
Loading

@NitzanMordhai there are still some leaks in the latest runs:

/a/yaarit-2026-03-05_02:43:32-rados-wip-rocky10-branch-of-the-day-2026-03-04-1772633736-distro-default-trial/86266/remote/trial015/log/valgrind/osd.0.log.gz

<error>
  <unique>0x1bf98</unique>
  <tid>1</tid>
  <kind>MismatchedFree</kind>
  <what>Mismatched free() / delete / delete []</what>
  <stack>
    <frame>
      <ip>0x61350CE</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete[](void*, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>1491</line>
    </frame>
    <frame>
      <ip>0x6DA65B0</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__run_exit_handlers</fn>
    </frame>
    <frame>
      <ip>0x6DA666F</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>exit</fn>
    </frame>
    <frame>
      <ip>0x6D8D594</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>(below main)</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x7674700 is 0 bytes inside a block of size 16,384 alloc'd</auxwhat>
  <stack>
    <frame>
      <ip>0x612F732</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>547</line>
    </frame>
    <frame>
      <ip>0x452A487</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>_sub_I_65535_1.1</fn>
    </frame>
    <frame>
      <ip>0x6D8D6BD</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__libc_start_main@@GLIBC_2.34</fn>
    </frame>
    <frame>
      <ip>0x452BA24</ip>
      <obj>/usr/bin/ceph-osd-classic</obj>
      <fn>(below main)</fn>
    </frame>
  </stack>
  <suppression>
    <sname>insert_a_suppression_name_here</sname>
    <skind>Memcheck:Free</skind>
    <sframe> <fun>_ZdaPvSt11align_val_t</fun> </sframe>
    <sframe> <fun>__run_exit_handlers</fun> </sframe>
    <sframe> <fun>exit</fun> </sframe>
    <sframe> <fun>(below main)</fun> </sframe>
    <rawtext>
<![CDATA[
{
   <insert_a_suppression_name_here>
   Memcheck:Free
   fun:_ZdaPvSt11align_val_t
   fun:__run_exit_handlers
   fun:exit
   fun:(below main)
}
]]>
    </rawtext>
  </suppression>
</error>

  <suppression>
    <sname>insert_a_suppression_name_here</sname>
    <skind>Memcheck:Free</skind>
    <sframe> <fun>_ZdaPvSt11align_val_t</fun> </sframe>
    <sframe> <fun>__run_exit_handlers</fun> </sframe>
    <sframe> <fun>exit</fun> </sframe>
    <sframe> <fun>(below main)</fun> </sframe>
    <rawtext>
<![CDATA[
{
   <insert_a_suppression_name_here>
   Memcheck:Free
   fun:_ZdaPvSt11align_val_t
   fun:__run_exit_handlers
   fun:exit
   fun:(below main)
}
]]>

/a/yaarit-2026-03-05_02:43:32-rados-wip-rocky10-branch-of-the-day-2026-03-04-1772633736-distro-default-trial/86266/remote/trial015/log/valgrind/mon.a.log.gz

 <unique>0x30494</unique>
  <tid>1</tid>
  <threadname>ceph-mon</threadname>
  <kind>MismatchedFree</kind>
  <what>Mismatched free() / delete / delete []</what>
  <stack>
    <frame>
      <ip>0x53A50CE</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator delete[](void*, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>1491</line>
    </frame>
    <frame>
      <ip>0x6402070</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__cxa_finalize</fn>
    </frame>
    <frame>
      <ip>0x5602966</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
    </frame>
    <frame>
      <ip>0x4B5D0E1</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
      <fn>_dl_call_fini</fn>
      <dir>/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/elf</dir>
      <file>dl-call_fini.c</file>
      <line>43</line>
    </frame>
    <frame>
      <ip>0x4B612EF</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
      <fn>_dl_fini</fn>
      <dir>/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/elf</dir>
      <file>dl-fini.c</file>
      <line>120</line>
    </frame>
    <frame>
      <ip>0x64025B0</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>__run_exit_handlers</fn>
    </frame>
    <frame>
      <ip>0x640266F</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>exit</fn>
    </frame>
    <frame>
      <ip>0x63E9594</ip>
      <obj>/usr/lib64/libc.so.6</obj>
      <fn>(below main)</fn>
    </frame>
  </stack>
  <auxwhat>Address 0x7083f00 is 0 bytes inside a block of size 16,384 alloc'd</auxwhat>
  <stack>
    <frame>
      <ip>0x539F732</ip>
      <obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
      <fn>operator new(unsigned long, std::align_val_t)</fn>
      <dir>/builddir/build/BUILD/valgrind-3.25.1/coregrind/m_replacemalloc</dir>
      <file>vg_replace_malloc.c</file>
      <line>547</line>
    </frame>
    <frame>
      <ip>0x5601D05</ip>
      <obj>/usr/lib64/ceph/libceph-common.so.2</obj>
      <fn>_sub_I_65535_1.1</fn>
    </frame>
    <frame>
      <ip>0x4B614CD</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
      <fn>call_init</fn>
      <dir>/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/elf</dir>
      <file>dl-init.c</file>
      <line>74</line>
    </frame>
    <frame>
      <ip>0x4B614CD</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
      <fn>call_init</fn>
      <dir>/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/elf</dir>
      <file>dl-init.c</file>
      <line>26</line>
    </frame>
    <frame>
      <ip>0x4B615BB</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
      <fn>_dl_init</fn>
      <dir>/usr/src/debug/glibc-2.39-58.el10_1.7.x86_64/elf</dir>
      <file>dl-init.c</file>
      <line>121</line>
    </frame>
    <frame>
      <ip>0x4B7949F</ip>
      <obj>/usr/lib64/ld-linux-x86-64.so.2</obj>
    </frame>
    <frame>
      <ip>0x5</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BDE</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BE7</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BEA</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BF4</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BF9</ip>
    </frame>
    <frame>
      <ip>0x1FFF000BFC</ip>
    </frame>
  </stack>
  <suppression>
    <sname>insert_a_suppression_name_here</sname>
    <skind>Memcheck:Free</skind>
    <sframe> <fun>_ZdaPvSt11align_val_t</fun> </sframe>
    <sframe> <fun>__cxa_finalize</fun> </sframe>
    <sframe> <obj>/usr/lib64/ceph/libceph-common.so.2</obj> </sframe>
    <sframe> <fun>_dl_call_fini</fun> </sframe>
    <sframe> <fun>_dl_fini</fun> </sframe>
    <sframe> <fun>__run_exit_handlers</fun> </sframe>
    <sframe> <fun>exit</fun> </sframe>
    <sframe> <fun>(below main)</fun> </sframe>
    <rawtext>
<![CDATA[
{
   <insert_a_suppression_name_here>
   Memcheck:Free
   fun:_ZdaPvSt11align_val_t
   fun:__cxa_finalize
   obj:/usr/lib64/ceph/libceph-common.so.2
   fun:_dl_call_fini
   fun:_dl_fini
   fun:__run_exit_handlers
   fun:exit
   fun:(below main)
}

@batrick
Copy link
Member

batrick commented Mar 6, 2026

This PR is under test in https://tracker.ceph.com/issues/75377.

@NitzanMordhai
qa: suppress false positive delete map mismatch errors
ae9ae03 
Valgrind reports "Mismatched free() / delete / delete []" errors during
OSD startup.

Standard library containers (like std::map) correctly call delete, but
Valgrind falsely interprets this as a call to delete[] because GCC 14
folds the identical aligned delete operators into a single symbol. This
causes Valgrind to flag a mismatch against the non-array allocation.

Fixes: https://tracker.ceph.com/issues/74604
Signed-off-by: Nitzan Mordechai <nmordech@ibm.com>
@NitzanMordhai NitzanMordhai force-pushed the wip-nitzan-rocky10-valgrind-mismatch-delete-map-gcc14 branch from ac1b45b to ae9ae03 Compare March 8, 2026 09:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@athanatos athanatos athanatos approved these changes

@rzarzynski rzarzynski rzarzynski approved these changes

Assignees

No one assigned

Labels

core needs-qa rocky10 tests ubuntu-24.04 wip-pdonnell-testing2

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@NitzanMordhai @ljflores @batrick @athanatos @rzarzynski @yaarith