Bug #52136
closedValgrind reports memory "Leak_DefinitelyLost" errors.
0%
Description
Valgrind reported the memory leak error in the following jobs:
/a/yuriw-2021-08-05_21:11:40-rados-wip-yuri-testing-2021-08-05-0936-distro-basic-smithi/6315879
/a/yuriw-2021-08-05_21:11:40-rados-wip-yuri-testing-2021-08-05-0936-distro-basic-smithi/6316099
/a/yuriw-2021-08-10_20:56:51-rados-wip-yuri-testing-2021-08-10-1139-distro-basic-smithi/6332241
Following is the error reported from 6315879 on osd.0:
<error>
<unique>0x51c2a</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>8 bytes in 1 blocks are definitely lost in loss record 140 of 3,017</text>
<leakedbytes>8</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4C31C93</ip>
<obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new[](unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.16.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>431</line>
</frame>
<frame>
<ip>0x1B9EF17</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionStorageInfo::VersionStorageInfo(rocksdb::InternalKeyComparator const*, rocksdb::Comparator const*, int, rocksdb::CompactionStyle, rocksdb::VersionStorageInfo*, bool)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>1724</line>
</frame>
<frame>
<ip>0x1B9F685</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::Version::Version(rocksdb::ColumnFamilyData*, rocksdb::VersionSet*, rocksdb::FileOptions const&, rocksdb::MutableCFOptions, std::shared_ptr<rocksdb::IOTracer> const&, unsigned long)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>1791</line>
</frame>
<frame>
<ip>0x1BB55AD</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionSet::CreateColumnFamily(rocksdb::ColumnFamilyOptions const&, rocksdb::VersionEdit const*)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>5764</line>
</frame>
<frame>
<ip>0x1E095F5</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandler::CreateCfAndInit(rocksdb::ColumnFamilyOptions const&, rocksdb::VersionEdit const&)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>409</line>
</frame>
<frame>
<ip>0x1E07BDC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandler::Initialize()</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>143</line>
</frame>
<frame>
<ip>0x1E06F53</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandlerBase::Iterate(rocksdb::log::Reader&, rocksdb::Status*)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>26</line>
</frame>
<frame>
<ip>0x1BAF7AC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>4752</line>
</frame>
<frame>
<ip>0x1ACCFE7</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>473</line>
</frame>
<frame>
<ip>0x1AD3996</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>1554</line>
</frame>
<frame>
<ip>0x1AD2D03</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>1448</line>
</frame>
<frame>
<ip>0x19D7E4A</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/kv</dir>
<file>RocksDBStore.cc</file>
<line>1152</line>
</frame>
<frame>
<ip>0x19EFA07</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>RocksDBStore::open(std::ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/kv</dir>
<file>RocksDBStore.h</file>
<line>258</line>
</frame>
<frame>
<ip>0x138573D</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_open_db(bool, bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>6253</line>
</frame>
<frame>
<ip>0x13FBDFD</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_open_db_and_around(bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>6012</line>
</frame>
<frame>
<ip>0x13FE495</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_mount()</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>7171</line>
</frame>
<frame>
<ip>0x14334CE</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::mount()</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/os/bluestore</dir>
<file>BlueStore.h</file>
<line>2623</line>
</frame>
<frame>
<ip>0xE3AB1D</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>OSD::init()</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src/osd</dir>
<file>OSD.cc</file>
<line>3554</line>
</frame>
<frame>
<ip>0xDB1BC2</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>main</fn>
<dir>/usr/src/debug/ceph-17.0.0-6738.g5d27ebc9.el8.x86_64/src</dir>
<file>ceph_osd.cc</file>
<line>711</line>
</frame>
</stack>
</error>
Updated by Adam Kupczyk over 4 years ago
This leak is from internals of RocksDB.
We have no access to FileMetaData objects, we cannot be responsible for this leak.
We should either forward this to RocksDB developers, or ignore (my preferred option).
Updated by Neha Ojha over 4 years ago
Adam Kupczyk wrote:
This leak is from internals of RocksDB.
We have no access to FileMetaData objects, we cannot be responsible for this leak.
We should either forward this to RocksDB developers, or ignore (my preferred option).
we could add it to the valgrind suppression file.
Updated by Laura Flores over 4 years ago
/a/yuriw-2021-12-09_00:18:57-rados-wip-yuri-testing-2021-12-08-1336-distro-default-smithi/6553724/ ----> osd.1.log.gz
Updated by Laura Flores about 4 years ago
/a/sseshasa-2022-02-24_11:27:07-rados-wip-45118-45121-quincy-testing-distro-default-smithi/6704275/remote/smithi174/log/valgrind/osd.6.log.gz
Updated by Neha Ojha about 4 years ago
Let's add it to qa/valgrind.supp to suppress this error, based on Adam's comment https://tracker.ceph.com/issues/52136#note-1.
Updated by Laura Flores almost 4 years ago
/a/yuriw-2022-03-26_19:43:35-rados-wip-yuri7-testing-2022-03-24-1341-pacific-distro-default-smithi/6762662
Updated by Kamoltat (Junior) Sirivadhna over 3 years ago
/a/yuriw-2022-08-08_22:19:17-rados-wip-yuri-testing-2022-08-08-1230-quincy-distro-default-smithi/6962388/
Updated by Radoslaw Zarzynski over 3 years ago
How about adding to suppression file regarding the comment #1?
Updated by Nitzan Mordechai over 3 years ago
- Related to Bug #57751: LibRadosAio.SimpleWritePP hang and pkill added
Updated by Nitzan Mordechai over 3 years ago
- Status changed from New to In Progress
Updated by Neha Ojha over 3 years ago
- Status changed from In Progress to Fix Under Review
- Pull request ID set to 48641
Updated by Nitzan Mordechai over 3 years ago
- Related to Bug #57618: rados/test.sh hang and pkilled (LibRadosWatchNotifyEC.WatchNotify) added
Updated by Yuri Weinstein over 3 years ago
Updated by Radoslaw Zarzynski over 3 years ago
- Status changed from Fix Under Review to Pending Backport
- Backport set to pacific,quincy
Updated by Upkeep Bot over 3 years ago
- Copied to Backport #58314: pacific: Valgrind reports memory "Leak_DefinitelyLost" errors. added
Updated by Upkeep Bot over 3 years ago
- Copied to Backport #58315: quincy: Valgrind reports memory "Leak_DefinitelyLost" errors. added
Updated by Nitzan Mordechai over 2 years ago
- Status changed from Pending Backport to Resolved
Updated by Shraddha Agrawal over 1 year ago
Found this error in /a/skanta-2024-10-05_10:36:55-rados-wip-bharath15-testing-2024-10-05-1105-quincy-distro-default-smithi/7934382.
<error>
<unique>0x93c0f</unique>
<tid>1</tid>
<kind>Leak_DefinitelyLost</kind>
<xwhat>
<text>8 bytes in 1 blocks are definitely lost in loss record 178 of 6,069</text>
<leakedbytes>8</leakedbytes>
<leakedblocks>1</leakedblocks>
</xwhat>
<stack>
<frame>
<ip>0x4846743</ip>
<obj>/usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so</obj>
<fn>operator new[](unsigned long)</fn>
<dir>/builddir/build/BUILD/valgrind-3.23.0/coregrind/m_replacemalloc</dir>
<file>vg_replace_malloc.c</file>
<line>729</line>
</frame>
<frame>
<ip>0xF476EF</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionStorageInfo::VersionStorageInfo(rocksdb::InternalKeyComparator const*, rocksdb::Comparator const*, int, rocksdb::CompactionStyle, rocksdb::VersionStorageInfo*, bool)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>1724</line>
</frame>
<frame>
<ip>0xF476EF</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionStorageInfo::VersionStorageInfo(rocksdb::InternalKeyComparator const*, rocksdb::Comparator const*, int, rocksdb::CompactionStyle, rocksdb::VersionStorageInfo*, bool)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>1724</line>
</frame>
<frame>
<ip>0xF47CDA</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::Version::Version(rocksdb::ColumnFamilyData*, rocksdb::VersionSet*, rocksdb::FileOptions const&, rocksdb::MutableCFOptions, std::shared_ptr<rocksdb::IOTracer> const&, unsigned long)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>1772</line>
</frame>
<frame>
<ip>0xF5CB35</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionSet::CreateColumnFamily(rocksdb::ColumnFamilyOptions const&, rocksdb::VersionEdit const*)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>5764</line>
</frame>
<frame>
<ip>0x106CDC2</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandler::CreateCfAndInit(rocksdb::ColumnFamilyOptions const&, rocksdb::VersionEdit const&)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>409</line>
</frame>
<frame>
<ip>0x106DDB0</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandler::Initialize()</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>143</line>
</frame>
<frame>
<ip>0x105D34C</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionEditHandlerBase::Iterate(rocksdb::log::Reader&, rocksdb::Status*)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_edit_handler.cc</file>
<line>26</line>
</frame>
<frame>
<ip>0xF62926</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::VersionSet::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db</dir>
<file>version_set.cc</file>
<line>4752</line>
</frame>
<frame>
<ip>0xEE4B15</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>473</line>
</frame>
<frame>
<ip>0xEE7286</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>1554</line>
</frame>
<frame>
<ip>0xEE85D8</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/rocksdb/db/db_impl</dir>
<file>db_impl_open.cc</file>
<line>1448</line>
</frame>
<frame>
<ip>0xE51CDD</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/kv</dir>
<file>RocksDBStore.cc</file>
<line>1193</line>
</frame>
<frame>
<ip>0x9F9972</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_open_db(bool, bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>6724</line>
</frame>
<frame>
<ip>0x9FBBEB</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_open_db_and_around(bool, bool)</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>6430</line>
</frame>
<frame>
<ip>0xA03B5F</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>BlueStore::_mount()</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/os/bluestore</dir>
<file>BlueStore.cc</file>
<line>7720</line>
</frame>
<frame>
<ip>0x5ADCFB</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>OSD::init()</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src/osd</dir>
<file>OSD.cc</file>
<line>3660</line>
</frame>
<frame>
<ip>0x5067CC</ip>
<obj>/usr/bin/ceph-osd</obj>
<fn>main</fn>
<dir>/usr/src/debug/ceph-17.2.7-1693.g00fa28fd.el9.x86_64/src</dir>
<file>ceph_osd.cc</file>
<line>726</line>
</frame>
</stack>
</error>
Updated by Shraddha Agrawal over 1 year ago
- Related to Bug #68655: Valgrind reports memory "Leak_DefinitelyLost" errors. added
Updated by Shraddha Agrawal over 1 year ago
Created a new issue to track this: https://tracker.ceph.com/issues/68655.
Updated by Upkeep Bot 8 months ago
- Merge Commit set to a55987be8044c84718beac942f4f1582012f2964
- Fixed In set to v18.0.0-1508-ga55987be80
- Released In set to v18.2.0~799
- Upkeep Timestamp set to 2025-07-14T16:44:02+00:00