Ceph » rgw

@Soumya Koduri and @Jiffin Tony Thottan -- there is a theory that this may be related to a recent PR you merged that adds a thread to restore. We are asking you to revert that PR and see if that resolves this valgrind issue (and the related one listed above). Thank you!

J. Eric Ivancich wrote in #note-3:

@Soumya Koduri and @Jiffin Tony Thottan -- there is a theory that this may be related to a recent PR you merged that adds a thread to restore. We are asking you to revert that PR and see if that resolves this valgrind issue (and the related one listed above). Thank you!

the presence of neorados in this valgrind report is what made me suspect the restore changes from https://github.com/ceph/ceph/pull/62713

it's hard to pin down exactly when these failures started. and while Soumya's qa results from https://pulpito.ceph.com/soumyakoduri-2025-07-01_06:26:07-rgw-wip-skoduri-restore-glacier-distro-default-smithi/ don't show the leaks, the PR's merge of ~2 weeks ago seems plausible

to get better symbols, i reproduced with vstart using:

$ MON=1 OSD=1 RGW=1 MGR=0 MDS=0 ../src/vstart.sh -n -d --valgrind_rgw memcheck --valgrind_args '--leak-check=full --show-leak-kinds=all --log-file=valgrind.out --max-threads=1024 --suppressions=../qa/valgrind.supp'

there were a lot of leaks related to openssl; sharing only the Objecter ones:

==334166== 4 bytes in 1 blocks are indirectly lost in loss record 6 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xDDAA6DF: std::__new_allocator<int>::allocate(unsigned long, void const*) (new_allocator.h:151)
==334166==    by 0xDDA5BBB: allocate (allocator.h:203)
==334166==    by 0xDDA5BBB: allocate (alloc_traits.h:614)
==334166==    by 0xDDA5BBB: std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) (stl_vector.h:387)
==334166==    by 0xE3AB752: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1910)
==334166==    by 0xE399B25: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:238)
==334166==    by 0xE63AF69: Objecter::lookup_pg_mapping(pg_t const&, unsigned int, std::vector<int, std::allocator<int> >*, int*, std::vector<int, std::allocator<int> >*, int*) (Objecter.h:1762)
==334166==    by 0xE61363B: Objecter::_calc_target(Objecter::op_target_t*, Connection*, bool) (Objecter.cc:2937)
==334166==    by 0xE6017CE: Objecter::_linger_submit(Objecter::LingerOp*, ceph::shunique_lock<ceph::shared_mutex_debug>&) (Objecter.cc:900)
==334166==    by 0xE601459: Objecter::linger_watch(Objecter::LingerOp*, ObjectOperation&, SnapContext const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, ceph::buffer::v15_2_0::list&, boost::asio::any_completion_handler<void (boost::system::error_code, ceph::buffer::v15_2_0::list)>&&, unsigned long*) (Objecter.cc:862)
==334166==    by 0x87B2300: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1511)
==334166==    by 0x66E5F3A: auto neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}::operator()<boost::asio::detail::consign_handler<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >) const (RADOS.hpp:1627)
==334166==    by 0x66E60FF: void boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>::operator()<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object&&, neorados::IOContext&&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >&&) && (consign.hpp:82)

==334166== 4 bytes in 1 blocks are indirectly lost in loss record 7 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xDDAA6DF: std::__new_allocator<int>::allocate(unsigned long, void const*) (new_allocator.h:151)
==334166==    by 0xDDA5BBB: allocate (allocator.h:203)
==334166==    by 0xDDA5BBB: allocate (alloc_traits.h:614)
==334166==    by 0xDDA5BBB: std::_Vector_base<int, std::allocator<int> >::_M_allocate(unsigned long) (stl_vector.h:387)
==334166==    by 0xE3AB752: int* std::vector<int, std::allocator<int> >::_M_allocate_and_copy<__gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > > >(unsigned long, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >, __gnu_cxx::__normal_iterator<int const*, std::vector<int, std::allocator<int> > >) (stl_vector.h:1910)
==334166==    by 0xE399B25: std::vector<int, std::allocator<int> >::operator=(std::vector<int, std::allocator<int> > const&) (vector.tcc:238)
==334166==    by 0xE63AF8D: Objecter::lookup_pg_mapping(pg_t const&, unsigned int, std::vector<int, std::allocator<int> >*, int*, std::vector<int, std::allocator<int> >*, int*) (Objecter.h:1764)
==334166==    by 0xE61363B: Objecter::_calc_target(Objecter::op_target_t*, Connection*, bool) (Objecter.cc:2937)
==334166==    by 0xE6017CE: Objecter::_linger_submit(Objecter::LingerOp*, ceph::shunique_lock<ceph::shared_mutex_debug>&) (Objecter.cc:900)
==334166==    by 0xE601459: Objecter::linger_watch(Objecter::LingerOp*, ObjectOperation&, SnapContext const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, ceph::buffer::v15_2_0::list&, boost::asio::any_completion_handler<void (boost::system::error_code, ceph::buffer::v15_2_0::list)>&&, unsigned long*) (Objecter.cc:862)
==334166==    by 0x87B2300: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1511)
==334166==    by 0x66E5F3A: auto neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}::operator()<boost::asio::detail::consign_handler<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >) const (RADOS.hpp:1627)
==334166==    by 0x66E60FF: void boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>::operator()<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object&&, neorados::IOContext&&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >&&) && (consign.hpp:82)

==334166== 24 bytes in 1 blocks are indirectly lost in loss record 535 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xDC65912: std::__new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:151)
==334166==    by 0xDC63A80: allocate (allocator.h:203)
==334166==    by 0xDC63A80: allocate (alloc_traits.h:614)
==334166==    by 0xDC63A80: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_allocate(std::allocator<char>&, unsigned long) (basic_string.h:142)
==334166==    by 0xDC63961: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) (basic_string.tcc:164)
==334166==    by 0xDC65403: void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char const*>(char const*, char const*, std::forward_iterator_tag) (basic_string.tcc:235)
==334166==    by 0xDC7571A: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(char const*, unsigned long, std::allocator<char> const&) (basic_string.h:692)
==334166==    by 0xE8A619B: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > fmt::v11::to_string<500ul>(fmt::v11::basic_memory_buffer<char, 500ul, fmt::v11::detail::allocator<char> > const&) (format.h:886)
==334166==    by 0xE8A30BA: fmt::v11::vformat[abi:cxx11](fmt::v11::basic_string_view<char>, fmt::v11::basic_format_args<fmt::v11::context>) (format-inl.h:1447)
==334166==    by 0xE299905: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > fmt::v11::format<unsigned long&>(fmt::v11::fstring<unsigned long&>::t, unsigned long&) (format.h:4200)
==334166==    by 0xE6294D1: Objecter::LingerOp::LingerOp(Objecter*, unsigned long) (Objecter.cc:4981)
==334166==    by 0xE600F93: Objecter::linger_register(object_t const&, object_locator_t const&, int) (Objecter.cc:823)
==334166==    by 0x87B2119: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1503)

==334166== 31 bytes in 1 blocks are indirectly lost in loss record 537 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xDC65912: std::__new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:151)
==334166==    by 0xDC63A80: allocate (allocator.h:203)
==334166==    by 0xDC63A80: allocate (alloc_traits.h:614)
==334166==    by 0xDC63A80: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_allocate(std::allocator<char>&, unsigned long) (basic_string.h:142)
==334166==    by 0xDC63961: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) (basic_string.tcc:164)
==334166==    by 0xDC84A87: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.tcc:319)
==334166==    by 0xDC97D17: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.h:1771)
==334166==    by 0xDC97AC2: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.h:906)
==334166==    by 0xDEC6578: object_t::operator=(object_t const&) (object.h:36)
==334166==    by 0xE600FCD: Objecter::linger_register(object_t const&, object_locator_t const&, int) (Objecter.cc:824)
==334166==    by 0x87B2119: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1503)
==334166==    by 0x66E5F3A: auto neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}::operator()<boost::asio::detail::consign_handler<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >) const (RADOS.hpp:1627)
==334166==    by 0x66E60FF: void boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>::operator()<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object&&, neorados::IOContext&&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >&&) && (consign.hpp:82)

==334166== 31 bytes in 1 blocks are indirectly lost in loss record 538 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xDC65912: std::__new_allocator<char>::allocate(unsigned long, void const*) (new_allocator.h:151)
==334166==    by 0xDC63A80: allocate (allocator.h:203)
==334166==    by 0xDC63A80: allocate (alloc_traits.h:614)
==334166==    by 0xDC63A80: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_allocate(std::allocator<char>&, unsigned long) (basic_string.h:142)
==334166==    by 0xDC63961: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) (basic_string.tcc:164)
==334166==    by 0xDC84A87: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.tcc:319)
==334166==    by 0xDC97D17: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.h:1771)
==334166==    by 0xDC97AC2: std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::operator=(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (basic_string.h:906)
==334166==    by 0xDEC6578: object_t::operator=(object_t const&) (object.h:36)
==334166==    by 0xE612FB8: Objecter::_calc_target(Objecter::op_target_t*, Connection*, bool) (Objecter.cc:2894)
==334166==    by 0xE6017CE: Objecter::_linger_submit(Objecter::LingerOp*, ceph::shunique_lock<ceph::shared_mutex_debug>&) (Objecter.cc:900)
==334166==    by 0xE601459: Objecter::linger_watch(Objecter::LingerOp*, ObjectOperation&, SnapContext const&, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >, ceph::buffer::v15_2_0::list&, boost::asio::any_completion_handler<void (boost::system::error_code, ceph::buffer::v15_2_0::list)>&&, unsigned long*) (Objecter.cc:862)
==334166==    by 0x87B2300: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1511)

==334166== 1,214 (1,120 direct, 94 indirect) bytes in 1 blocks are definitely lost in loss record 787 of 826
==334166==    at 0xD7F3A3A: operator new[](unsigned long) (vg_replace_malloc.c:729)
==334166==    by 0xE600F4A: Objecter::linger_register(object_t const&, object_locator_t const&, int) (Objecter.cc:823)
==334166==    by 0x87B2119: neorados::RADOS::watch_(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::any_completion_handler<void (boost::system::error_code, unsigned long)>) (RADOS.cc:1503)
==334166==    by 0x66E5F3A: auto neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}::operator()<boost::asio::detail::consign_handler<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >) const (RADOS.hpp:1627)
==334166==    by 0x66E60FF: void boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>::operator()<boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object&&, neorados::IOContext&&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >&&) && (consign.hpp:82)
==334166==    by 0x66E3331: boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::system::error_code, unsigned long>* boost::asio::async_result<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, void (boost::system::error_code, unsigned long)>::do_init<boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>*, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&, boost::asio::use_awaitable_t<boost::asio::any_io_executor>, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >&, neorados::Object&, neorados::IOContext&, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >&) (use_awaitable.hpp:209)
==334166==    by 0x66E0890: auto boost::asio::async_result<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, void (boost::system::error_code, unsigned long)>::initiate<boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> > >(boost::asio::async_result<boost::asio::consign_t<boost::asio::use_awaitable_t<boost::asio::any_io_executor>, boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, void (boost::system::error_code, unsigned long)>::init_wrapper<neorados::RADOS::watch<boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&>(neorados::Object, neorados::IOContext, std::optional<std::chrono::duration<long, std::ratio<1l, 1l> > >, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >, boost::asio::use_awaitable_t<boost::asio::any_io_executor> const&)::{lambda(auto:1&&, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)#1}>, boost::asio::use_awaitable_t<boost::asio::any_io_executor>, std::tuple<boost::asio::executor_work_guard<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul>, void, void> >, neorados::Object, neorados::IOContext, fu2::abi_400::detail::function<fu2::abi_400::detail::config<true, false, fu2::capacity_default>, fu2::abi_400::detail::property<true, false, void (boost::system::error_code, unsigned long, unsigned long, unsigned long, ceph::buffer::v15_2_0::list&&)> >)::{lambda(auto:1*)#1}::operator()<boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor> >(boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>*) const (use_awaitable.hpp:219)
==334166==    by 0x66E0936: _ZZZN5boost4asio6detail20awaitable_frame_baseINS0_15any_io_executorEE15await_transformIZNS0_12async_resultINS0_15use_awaitable_tIS3_EEJFvNS_6system10error_codeEmEEE8initiateINS6_INS0_9consign_tIS8_JNS0_19executor_work_guardINS0_10io_context19basic_executor_typeISaIvELm0EEEvvEEEEEJSB_EE12init_wrapperIZN8neorados5RADOS5watchIRKS8_EEDaNSO_6ObjectENSO_9IOContextESt8optionalINSt6chrono8durationIlSt5ratioILl1ELl1EEEEEN3fu27abi_4006detail8functionINS14_6configILb1ELb0ENS12_16capacity_defaultEEENS14_8propertyILb1ELb0EJFvSA_mmmON4ceph6buffer7v15_2_04listEEEEEEEOT_EUlS1J_ST_SU_S1H_E_EEJSt5tupleIJSK_EEST_SU_S1H_EEENS0_9awaitableImS3_EES1I_S8_DpT0_EUlPS1I_E_EEDaS1I_PNSt9enable_ifIXsrSt14is_convertibleINS0_9result_ofIFS1I_PS4_EE4typeEPNS1_16awaitable_threadIS3_EEE5valueEvE4typeEEN6result13await_suspendENSt7__n486116coroutine_handleIvEEENKUlPvE_clES2C_ (awaitable.hpp:424)
==334166==    by 0x66E0956: _ZZZN5boost4asio6detail20awaitable_frame_baseINS0_15any_io_executorEE15await_transformIZNS0_12async_resultINS0_15use_awaitable_tIS3_EEJFvNS_6system10error_codeEmEEE8initiateINS6_INS0_9consign_tIS8_JNS0_19executor_work_guardINS0_10io_context19basic_executor_typeISaIvELm0EEEvvEEEEEJSB_EE12init_wrapperIZN8neorados5RADOS5watchIRKS8_EEDaNSO_6ObjectENSO_9IOContextESt8optionalINSt6chrono8durationIlSt5ratioILl1ELl1EEEEEN3fu27abi_4006detail8functionINS14_6configILb1ELb0ENS12_16capacity_defaultEEENS14_8propertyILb1ELb0EJFvSA_mmmON4ceph6buffer7v15_2_04listEEEEEEEOT_EUlS1J_ST_SU_S1H_E_EEJSt5tupleIJSK_EEST_SU_S1H_EEENS0_9awaitableImS3_EES1I_S8_DpT0_EUlPS1I_E_EEDaS1I_PNSt9enable_ifIXsrSt14is_convertibleINS0_9result_ofIFS1I_PS4_EE4typeEPNS1_16awaitable_threadIS3_EEE5valueEvE4typeEEN6result13await_suspendENSt7__n486116coroutine_handleIvEEENUlPvE_4_FUNES2C_ (awaitable.hpp:421)
==334166==    by 0x40AF6A0: boost::asio::detail::awaitable_frame_base<boost::asio::any_io_executor>::resume() (awaitable.hpp:501)
==334166==    by 0x40ABA7C: boost::asio::detail::awaitable_thread<boost::asio::any_io_executor>::pump() (awaitable.hpp:769)
==334166==    by 0x66EAB57: void boost::asio::detail::awaitable_handler<boost::asio::any_io_executor, boost::container::flat_map<unsigned long, logback_generation, std::less<unsigned long>, void> >::operator()<boost::container::flat_map<unsigned long, logback_generation, std::less<unsigned long>, void> >(boost::container::flat_map<unsigned long, logback_generation, std::less<unsigned long>, void>&&) (use_awaitable.hpp:103)

@J. Eric Ivancich and @Casey Bodley ,

I reverted the commits from https://github.com/ceph/ceph/pull/62713 in a private branch - https://github.com/soumyakoduri/ceph/commits/wip-skoduri-valgrind/ and ran verify tests. Here are the results -

https://pulpito.ceph.com/soumyakoduri-2025-07-25_05:59:14-rgw:verify-wip-skoduri-valgrind-distro-default-smithi/

The valgrind warnings about `neorados::RADOS::watch_` are still seen even without these restore changes.