tools/cephfs: Creation of a CephFS Shell by rpavani1998 · Pull Request #5 · batrick/ceph

rpavani1998 · 2018-05-21T17:34:16Z

I have started implementing existing libcephfs client library and python bindings. As of now the shell allows basic commands like mkdir, ls, copyFromLocal , opendir,...

Signed-off-by: Pavani Rajula rpavani1998@gmail.com

batrick · 2018-05-22T00:21:41Z

doc/cephfs/shell.rst

+ -p        Creates parent directories if doesn't exist.
+
+
+copyFromLocal


let's use put.

Do you mean using a generic put command for both copyFromLocal and copyToLocal?

No, there would be a put and get (these are fairly well-known command names).

batrick · 2018-05-22T00:22:32Z

doc/cephfs/shell.rst

+
+ Copy a file to Ceph File System from Local Directory.
+
+ Usage : copyFromLocal [-option] <path_uri>


it should take a target path too. Write as:

put [options] <source_path> [target_path]

target_path is optional and defaults to /.

batrick · 2018-05-22T00:22:52Z

doc/cephfs/shell.rst

+ Options:
+
+ -s 	   List file size
+ -S 	   Sort by file size


batrick · 2018-05-22T00:23:39Z

doc/cephfs/shell.rst

+
+Open a specific named directory.
+
+Usage : opendir <directory_name>


What does it mean to open a directory in a shell? (Think about bash or sh, do these shells have a concept of opening a directory?)

It is actually cd command. I will change it to cd.

batrick · 2018-05-22T00:24:07Z

doc/cephfs/shell.rst

+
+Get current working directory.
+
+Usage : cwd


This is useful! Having th econcept of a current working directory is great. You will want a cd command too!

batrick · 2018-05-22T00:29:50Z

src/tools/cephfs/cephfs_shell.py

+    def do_mkdir(self,arglist):
+        """Create directory.
+            Usage: mkdir <new_dir> <permission> """    
+        parser = argparse.ArgumentParser(description='Create Directory.')


We won't want to recreate the parser everytime a shell command is run. It would be a good idea to pull this parser variable out into a static class member. You can read more about this here:

http://radek.io/2011/07/21/static-variables-and-methods-in-python/
https://stackoverflow.com/questions/872973/pythonic-way-to-initialize-complex-static-data-members

Using a static method to create the parser (one method for each command) seems reasonable to me. Then call the static method to initialize the parser for the command, named mkdir_parser. (There may be more "Pythonic" ways to solve this though.)

batrick · 2018-05-24T17:58:49Z

Odd, I don't know how this got closed. Please reopen a new PR @rpavani1998 . I'm not able to reopen this.

We just took the curmap ref above; do not call get_osdmap() again. I think it may explain a weird segv I saw here in ~shared_ptr, although I'm not quite certain. Regardless, this change is correct and better. (gdb) bt #0 raise (sig=sig@entry=11) at ../sysdeps/unix/sysv/linux/raise.c:51 #1 0x00005596e5a98261 in reraise_fatal (signum=11) at ./src/global/signal_handler.cc:326 #2 handle_fatal_signal(int) () at ./src/global/signal_handler.cc:326 #3 <signal handler called> #4 0x00005596f4fe80e0 in ?? () #5 0x00005596e5464068 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5596f4b7cf60) at /usr/include/c++/9/bits/shared_ptr_base.h:148 #6 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x5596f4b7cf60) at /usr/include/c++/9/bits/shared_ptr_base.h:148 #7 0x00005596e543377f in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=0x7f2b25044e28, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169 #8 std::__shared_ptr<OSDMap const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=0x7f2b25044e20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr_base.h:1169 ceph#9 std::shared_ptr<OSDMap const>::~shared_ptr (this=0x7f2b25044e20, __in_chrg=<optimized out>) at /usr/include/c++/9/bits/shared_ptr.h:103 ceph#10 OSD::handle_osd_ping(MOSDPing*) () at ./src/osd/OSD.cc:4662 Signed-off-by: Sage Weil <sage@redhat.com>

Accordingly to cppreference.com [1]: "If multiple threads of execution access the same std::shared_ptr object without synchronization and any of those accesses uses a non-const member function of shared_ptr then a data race will occur (...)" [1]: https://en.cppreference.com/w/cpp/memory/shared_ptr/atomic One of the coredumps showed the `shared_ptr`-typed `OSD::osdmap` with healthy looking content but damaged control block: ``` [Current thread is 1 (Thread 0x7f7dcaf73700 (LWP 205295))] (gdb) bt #0 0x0000559cb81c3ea0 in ?? () #1 0x0000559c97675b27 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 #3 0x0000559c975ef8aa in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1167 #4 std::__shared_ptr<OSDMap const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1167 #5 std::shared_ptr<OSDMap const>::~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr.h:103 #6 OSD::create_context (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9053 #7 0x0000559c97655571 in OSD::dequeue_peering_evt (this=0x559ca22ac000, sdata=0x559ca2ef2900, pg=0x559cb4aa3400, evt=std::shared_ptr<PGPeeringEvent> (use count 2, weak count 0) = {...}, handle=...) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9665 #8 0x0000559c97886db6 in ceph::osd::scheduler::PGPeeringItem::run (this=<optimized out>, osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at /usr/include/c++/8/ext/atomicity.h:96 ceph#9 0x0000559c9764862f in ceph::osd::scheduler::OpSchedulerItem::run (handle=..., pg=..., sdata=<optimized out>, osd=<optimized out>, this=0x7f7dcaf703f0) at /usr/include/c++/8/bits/unique_ptr.h:342 ceph#10 OSD::ShardedOpWQ::_process (this=<optimized out>, thread_index=<optimized out>, hb=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:10677 ceph#11 0x0000559c97c76094 in ShardedThreadPool::shardedthreadpool_worker (this=0x559ca22aca28, thread_index=14) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/common/WorkQueue.cc:311 ceph#12 0x0000559c97c78cf4 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/common/WorkQueue.h:706 ceph#13 0x00007f7df17852de in start_thread () from /lib64/libpthread.so.0 ceph#14 0x00007f7df052f133 in __libc_ifunc_impl_list () from /lib64/libc.so.6 ceph#15 0x0000000000000000 in ?? () (gdb) frame 7 #7 0x0000559c97655571 in OSD::dequeue_peering_evt (this=0x559ca22ac000, sdata=0x559ca2ef2900, pg=0x559cb4aa3400, evt=std::shared_ptr<PGPeeringEvent> (use count 2, weak count 0) = {...}, handle=...) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9665 9665 in /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc (gdb) print osdmap $24 = std::shared_ptr<const OSDMap> (expired, weak count 0) = {get() = 0x559cba028000} (gdb) print *osdmap # pretty sane OSDMap (gdb) print sizeof(osdmap) $26 = 16 (gdb) x/2a &osdmap 0x559ca22acef0: 0x559cba028000 0x559cba0ec900 (gdb) frame 2 #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 148 /usr/include/c++/8/bits/shared_ptr_base.h: No such file or directory. (gdb) disassemble Dump of assembler code for function std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(): ... 0x0000559c97675b1e <+62>: mov (%rdi),%rax 0x0000559c97675b21 <+65>: mov %rdi,%rbx 0x0000559c97675b24 <+68>: callq *0x10(%rax) => 0x0000559c97675b27 <+71>: test %rbp,%rbp ... End of assembler dump. (gdb) info registers rdi rbx rax rdi 0x559cba0ec900 94131624790272 rbx 0x559cba0ec900 94131624790272 rax 0x559cba0ec8a0 94131624790176 (gdb) x/a 0x559cba0ec8a0 + 0x10 0x559cba0ec8b0: 0x559cb81c3ea0 (gdb) bt #0 0x0000559cb81c3ea0 in ?? () ... (gdb) p $_siginfo._sifields._sigfault.si_addr $27 = (void *) 0x559cb81c3ea0 ``` Helgrind seems to agree: ``` ==00:00:02:54.519 510301== Possible data race during write of size 8 at 0xF123930 by thread ceph#90 ==00:00:02:54.519 510301== Locks held: 2, at addresses 0xF122A58 0xF1239A8 ==00:00:02:54.519 510301== at 0x7218DD: operator= (shared_ptr_base.h:1078) ==00:00:02:54.519 510301== by 0x7218DD: operator= (shared_ptr.h:103) ==00:00:02:54.519 510301== by 0x7218DD: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8116) ==00:00:02:54.519 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:02:54.519 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:02:54.519 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:02:54.519 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:02:54.519 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:02:54.519 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ==00:00:02:54.519 510301== ==00:00:02:54.519 510301== This conflicts with a previous read of size 8 by thread ceph#117 ==00:00:02:54.519 510301== Locks held: 1, at address 0x2123E9A0 ==00:00:02:54.519 510301== at 0x6B5842: __shared_ptr (shared_ptr_base.h:1165) ==00:00:02:54.519 510301== by 0x6B5842: shared_ptr (shared_ptr.h:129) ==00:00:02:54.519 510301== by 0x6B5842: get_osdmap (OSD.h:1700) ==00:00:02:54.519 510301== by 0x6B5842: OSD::create_context() (OSD.cc:9053) ==00:00:02:54.519 510301== by 0x71B570: OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&) (OSD.cc:9665) ==00:00:02:54.519 510301== by 0x71B997: OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&) (OSD.cc:9701) ==00:00:02:54.519 510301== by 0x70E62E: run (OpSchedulerItem.h:148) ==00:00:02:54.519 510301== by 0x70E62E: OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (OSD.cc:10677) ==00:00:02:54.519 510301== by 0xD3C093: ShardedThreadPool::shardedthreadpool_worker(unsigned int) (WorkQueue.cc:311) ==00:00:02:54.519 510301== by 0xD3ECF3: ShardedThreadPool::WorkThreadSharded::entry() (WorkQueue.h:706) ==00:00:02:54.519 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:02:54.519 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:02:54.519 510301== Address 0xf123930 is 3,824 bytes inside a block of size 10,296 alloc'd ==00:00:02:54.519 510301== at 0xA7DC0C3: operator new[](unsigned long) (vg_replace_malloc.c:433) ==00:00:02:54.519 510301== by 0x66F766: main (ceph_osd.cc:688) ==00:00:02:54.519 510301== Block was alloc'd by thread #1 ``` Actually there is plenty of similar issues reported like: ``` ==00:00:05:04.903 510301== Possible data race during read of size 8 at 0x1E3E0588 by thread ceph#119 ==00:00:05:04.903 510301== Locks held: 1, at address 0x1EAD41D0 ==00:00:05:04.903 510301== at 0x753165: clear (hashtable.h:2051) ==00:00:05:04.903 510301== by 0x753165: std::_Hashtable<entity_addr_t, std::pair<entity_addr_t const, utime_t>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<entity_addr_t const, utime_t> >, std::__detail::_Select1st, std::equal_to<entity_addr_t>, std::hash<entity_addr_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__deta il::_Hashtable_traits<true, false, true> >::~_Hashtable() (hashtable.h:1369) ==00:00:05:04.903 510301== by 0x75331C: ~unordered_map (unordered_map.h:102) ==00:00:05:04.903 510301== by 0x75331C: OSDMap::~OSDMap() (OSDMap.h:350) ==00:00:05:04.903 510301== by 0x753606: operator() (shared_cache.hpp:100) ==00:00:05:04.903 510301== by 0x753606: std::_Sp_counted_deleter<OSDMap const*, SharedLRU<unsigned int, OSDMap const>::Cleanup, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr _base.h:471) ==00:00:05:04.903 510301== by 0x73BB26: _M_release (shared_ptr_base.h:155) ==00:00:05:04.903 510301== by 0x73BB26: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148) ==00:00:05:04.903 510301== by 0x6B58A9: ~__shared_count (shared_ptr_base.h:728) ==00:00:05:04.903 510301== by 0x6B58A9: ~__shared_ptr (shared_ptr_base.h:1167) ==00:00:05:04.903 510301== by 0x6B58A9: ~shared_ptr (shared_ptr.h:103) ==00:00:05:04.903 510301== by 0x6B58A9: OSD::create_context() (OSD.cc:9053) ==00:00:05:04.903 510301== by 0x71B570: OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&) (OSD.cc:9665) ==00:00:05:04.903 510301== by 0x71B997: OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&) (OSD.cc:9701) ==00:00:05:04.903 510301== by 0x70E62E: run (OpSchedulerItem.h:148) ==00:00:05:04.903 510301== by 0x70E62E: OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (OSD.cc:10677) ==00:00:05:04.903 510301== by 0xD3C093: ShardedThreadPool::shardedthreadpool_worker(unsigned int) (WorkQueue.cc:311) ==00:00:05:04.903 510301== by 0xD3ECF3: ShardedThreadPool::WorkThreadSharded::entry() (WorkQueue.h:706) ==00:00:05:04.903 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:05:04.903 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:05:04.903 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ==00:00:05:04.903 510301== ==00:00:05:04.903 510301== This conflicts with a previous write of size 8 by thread ceph#90 ==00:00:05:04.903 510301== Locks held: 2, at addresses 0xF122A58 0xF1239A8 ==00:00:05:04.903 510301== at 0x7531E1: clear (hashtable.h:2054) ==00:00:05:04.903 510301== by 0x7531E1: std::_Hashtable<entity_addr_t, std::pair<entity_addr_t const, utime_t>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<entity_addr_t const, utime_t> >, std::__detail::_Select1st, std::equal_to<entity_addr_t>, std::hash<entity_addr_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable() (hashtable.h:1369) ==00:00:05:04.903 510301== by 0x75331C: ~unordered_map (unordered_map.h:102) ==00:00:05:04.903 510301== by 0x75331C: OSDMap::~OSDMap() (OSDMap.h:350) ==00:00:05:04.903 510301== by 0x753606: operator() (shared_cache.hpp:100) ==00:00:05:04.903 510301== by 0x753606: std::_Sp_counted_deleter<OSDMap const*, SharedLRU<unsigned int, OSDMap const>::Cleanup, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:471) ==00:00:05:04.903 510301== by 0x73BB26: _M_release (shared_ptr_base.h:155) ==00:00:05:04.903 510301== by 0x73BB26: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr_base.h:747) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr_base.h:1078) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr.h:103) ==00:00:05:04.903 510301== by 0x72191E: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8116) ==00:00:05:04.903 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:05:04.903 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:05:04.903 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:05:04.903 510301== Address 0x1e3e0588 is 872 bytes inside a block of size 1,208 alloc'd ==00:00:05:04.903 510301== at 0xA7DC0C3: operator new[](unsigned long) (vg_replace_malloc.c:433) ==00:00:05:04.903 510301== by 0x6C7C0C: OSDService::try_get_map(unsigned int) (OSD.cc:1606) ==00:00:05:04.903 510301== by 0x7213BD: get_map (OSD.h:699) ==00:00:05:04.903 510301== by 0x7213BD: get_map (OSD.h:1732) ==00:00:05:04.903 510301== by 0x7213BD: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8076) ==00:00:05:04.903 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:05:04.903 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:05:04.903 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:05:04.903 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:05:04.903 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:05:04.903 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

* no need to discard_result(). as `output_stream::close()` returns an empty future<> already * free the connected socket after the background task finishes, because: we should not free the connected socket before the promise referencing it is fulfilled. otherwise we have error messages from ASan, like ==287182==ERROR: AddressSanitizer: heap-use-after-free on address 0x611000019aa0 at pc 0x55e2ae2de882 bp 0x7fff7e2bf080 sp 0x7fff7e2bf078 READ of size 8 at 0x611000019aa0 thread T0 #0 0x55e2ae2de881 in seastar::reactor_backend_aio::await_events(int, __sigset_t const*) ../src/seastar/src/core/reactor_backend.cc:396 #1 0x55e2ae2dfb59 in seastar::reactor_backend_aio::reap_kernel_completions() ../src/seastar/src/core/reactor_backend.cc:428 #2 0x55e2adbea397 in seastar::reactor::reap_kernel_completions_pollfn::poll() (/var/ssd/ceph/build/bin/crimson-osd+0x155e9397) #3 0x55e2adaec6d0 in seastar::reactor::poll_once() ../src/seastar/src/core/reactor.cc:2789 #4 0x55e2adae7cf7 in operator() ../src/seastar/src/core/reactor.cc:2687 #5 0x55e2adb7c595 in __invoke_impl<bool, seastar::reactor::run()::<lambda()>&> /usr/include/c++/10/bits/invoke.h:60 #6 0x55e2adb699b0 in __invoke_r<bool, seastar::reactor::run()::<lambda()>&> /usr/include/c++/10/bits/invoke.h:113 #7 0x55e2adb50222 in _M_invoke /usr/include/c++/10/bits/std_function.h:291 #8 0x55e2adc2ba00 in std::function<bool ()>::operator()() const /usr/include/c++/10/bits/std_function.h:622 ceph#9 0x55e2adaea491 in seastar::reactor::run() ../src/seastar/src/core/reactor.cc:2713 ceph#10 0x55e2ad98f1c7 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) ../src/seastar/src/core/app-template.cc:199 ceph#11 0x55e2a9e57538 in main ../src/crimson/osd/main.cc:148 ceph#12 0x7fae7f20de0a in __libc_start_main ../csu/libc-start.c:308 ceph#13 0x55e2a9d431e9 in _start (/var/ssd/ceph/build/bin/crimson-osd+0x117421e9) 0x611000019aa0 is located 96 bytes inside of 240-byte region [0x611000019a40,0x611000019b30) freed by thread T0 here: #0 0x7fae80a4e487 in operator delete(void*, unsigned long) (/usr/lib/x86_64-linux-gnu/libasan.so.6+0xac487) #1 0x55e2ae302a0a in seastar::aio_pollable_fd_state::~aio_pollable_fd_state() ../src/seastar/src/core/reactor_backend.cc:458 #2 0x55e2ae2e1059 in seastar::reactor_backend_aio::forget(seastar::pollable_fd_state&) ../src/seastar/src/core/reactor_backend.cc:524 #3 0x55e2adab9b9a in seastar::pollable_fd_state::forget() ../src/seastar/src/core/reactor.cc:1396 #4 0x55e2adab9d05 in seastar::intrusive_ptr_release(seastar::pollable_fd_state*) ../src/seastar/src/core/reactor.cc:1401 #5 0x55e2ace1b72b in boost::intrusive_ptr<seastar::pollable_fd_state>::~intrusive_ptr() /opt/ceph/include/boost/smart_ptr/intrusive_ptr.hpp:98 #6 0x55e2ace115a5 in seastar::pollable_fd::~pollable_fd() ../src/seastar/include/seastar/core/internal/pollable_fd.hh:109 #7 0x55e2ae0ed35c in seastar::net::posix_server_socket_impl::~posix_server_socket_impl() ../src/seastar/include/seastar/net/posix-stack.hh:161 #8 0x55e2ae0ed3cf in seastar::net::posix_server_socket_impl::~posix_server_socket_impl() ../src/seastar/include/seastar/net/posix-stack.hh:161 ceph#9 0x55e2ae0ed943 in std::default_delete<seastar::net::api_v2::server_socket_impl>::operator()(seastar::net::api_v2::server_socket_impl*) const /usr/include/c++/10/bits/unique_ptr.h:81 ceph#10 0x55e2ae0db357 in std::unique_ptr<seastar::net::api_v2::server_socket_impl, std::default_delete<seastar::net::api_v2::server_socket_impl> >::~unique_ptr() /usr/include/c++/10/bits/unique_ptr.h:357 ceph#11 0x55e2ae1438b7 in seastar::api_v2::server_socket::~server_socket() ../src/seastar/src/net/stack.cc:195 ceph#12 0x55e2aa1c7656 in std::_Optional_payload_base<seastar::api_v2::server_socket>::_M_destroy() /usr/include/c++/10/optional:260 ceph#13 0x55e2aa16c84b in std::_Optional_payload_base<seastar::api_v2::server_socket>::_M_reset() /usr/include/c++/10/optional:280 ceph#14 0x55e2ac24b2b7 in std::_Optional_base_impl<seastar::api_v2::server_socket, std::_Optional_base<seastar::api_v2::server_socket, false, false> >::_M_reset() /usr/include/c++/10/optional:432 ceph#15 0x55e2ac23f37b in std::optional<seastar::api_v2::server_socket>::reset() /usr/include/c++/10/optional:975 ceph#16 0x55e2ac21a2e7 in crimson::admin::AdminSocket::stop() ../src/crimson/admin/admin_socket.cc:265 ceph#17 0x55e2aa099825 in operator() ../src/crimson/osd/osd.cc:450 ceph#18 0x55e2aa0d4e3e in apply ../src/seastar/include/seastar/core/apply.hh:36 Signed-off-by: Kefu Chai <kchai@redhat.com>

Accordingly to cppreference.com [1]: "If multiple threads of execution access the same std::shared_ptr object without synchronization and any of those accesses uses a non-const member function of shared_ptr then a data race will occur (...)" [1]: https://en.cppreference.com/w/cpp/memory/shared_ptr/atomic One of the coredumps showed the `shared_ptr`-typed `OSD::osdmap` with healthy looking content but damaged control block: ``` [Current thread is 1 (Thread 0x7f7dcaf73700 (LWP 205295))] (gdb) bt #0 0x0000559cb81c3ea0 in ?? () #1 0x0000559c97675b27 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 #3 0x0000559c975ef8aa in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1167 #4 std::__shared_ptr<OSDMap const, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr_base.h:1167 #5 std::shared_ptr<OSDMap const>::~shared_ptr (this=<optimized out>, __in_chrg=<optimized out>) at /usr/include/c++/8/bits/shared_ptr.h:103 #6 OSD::create_context (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9053 #7 0x0000559c97655571 in OSD::dequeue_peering_evt (this=0x559ca22ac000, sdata=0x559ca2ef2900, pg=0x559cb4aa3400, evt=std::shared_ptr<PGPeeringEvent> (use count 2, weak count 0) = {...}, handle=...) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9665 #8 0x0000559c97886db6 in ceph::osd::scheduler::PGPeeringItem::run (this=<optimized out>, osd=<optimized out>, sdata=<optimized out>, pg=..., handle=...) at /usr/include/c++/8/ext/atomicity.h:96 ceph#9 0x0000559c9764862f in ceph::osd::scheduler::OpSchedulerItem::run (handle=..., pg=..., sdata=<optimized out>, osd=<optimized out>, this=0x7f7dcaf703f0) at /usr/include/c++/8/bits/unique_ptr.h:342 ceph#10 OSD::ShardedOpWQ::_process (this=<optimized out>, thread_index=<optimized out>, hb=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:10677 ceph#11 0x0000559c97c76094 in ShardedThreadPool::shardedthreadpool_worker (this=0x559ca22aca28, thread_index=14) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/common/WorkQueue.cc:311 ceph#12 0x0000559c97c78cf4 in ShardedThreadPool::WorkThreadSharded::entry (this=<optimized out>) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/common/WorkQueue.h:706 ceph#13 0x00007f7df17852de in start_thread () from /lib64/libpthread.so.0 ceph#14 0x00007f7df052f133 in __libc_ifunc_impl_list () from /lib64/libc.so.6 ceph#15 0x0000000000000000 in ?? () (gdb) frame 7 #7 0x0000559c97655571 in OSD::dequeue_peering_evt (this=0x559ca22ac000, sdata=0x559ca2ef2900, pg=0x559cb4aa3400, evt=std::shared_ptr<PGPeeringEvent> (use count 2, weak count 0) = {...}, handle=...) at /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc:9665 9665 in /usr/src/debug/ceph-15.0.0-10071.g5b5a3a3.el8.x86_64/src/osd/OSD.cc (gdb) print osdmap $24 = std::shared_ptr<const OSDMap> (expired, weak count 0) = {get() = 0x559cba028000} (gdb) print *osdmap # pretty sane OSDMap (gdb) print sizeof(osdmap) $26 = 16 (gdb) x/2a &osdmap 0x559ca22acef0: 0x559cba028000 0x559cba0ec900 (gdb) frame 2 #2 std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release (this=0x559cba0ec900) at /usr/include/c++/8/bits/shared_ptr_base.h:148 148 /usr/include/c++/8/bits/shared_ptr_base.h: No such file or directory. (gdb) disassemble Dump of assembler code for function std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release(): ... 0x0000559c97675b1e <+62>: mov (%rdi),%rax 0x0000559c97675b21 <+65>: mov %rdi,%rbx 0x0000559c97675b24 <+68>: callq *0x10(%rax) => 0x0000559c97675b27 <+71>: test %rbp,%rbp ... End of assembler dump. (gdb) info registers rdi rbx rax rdi 0x559cba0ec900 94131624790272 rbx 0x559cba0ec900 94131624790272 rax 0x559cba0ec8a0 94131624790176 (gdb) x/a 0x559cba0ec8a0 + 0x10 0x559cba0ec8b0: 0x559cb81c3ea0 (gdb) bt #0 0x0000559cb81c3ea0 in ?? () ... (gdb) p $_siginfo._sifields._sigfault.si_addr $27 = (void *) 0x559cb81c3ea0 ``` Helgrind seems to agree: ``` ==00:00:02:54.519 510301== Possible data race during write of size 8 at 0xF123930 by thread ceph#90 ==00:00:02:54.519 510301== Locks held: 2, at addresses 0xF122A58 0xF1239A8 ==00:00:02:54.519 510301== at 0x7218DD: operator= (shared_ptr_base.h:1078) ==00:00:02:54.519 510301== by 0x7218DD: operator= (shared_ptr.h:103) ==00:00:02:54.519 510301== by 0x7218DD: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8116) ==00:00:02:54.519 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:02:54.519 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:02:54.519 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:02:54.519 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:02:54.519 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:02:54.519 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ==00:00:02:54.519 510301== ==00:00:02:54.519 510301== This conflicts with a previous read of size 8 by thread ceph#117 ==00:00:02:54.519 510301== Locks held: 1, at address 0x2123E9A0 ==00:00:02:54.519 510301== at 0x6B5842: __shared_ptr (shared_ptr_base.h:1165) ==00:00:02:54.519 510301== by 0x6B5842: shared_ptr (shared_ptr.h:129) ==00:00:02:54.519 510301== by 0x6B5842: get_osdmap (OSD.h:1700) ==00:00:02:54.519 510301== by 0x6B5842: OSD::create_context() (OSD.cc:9053) ==00:00:02:54.519 510301== by 0x71B570: OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&) (OSD.cc:9665) ==00:00:02:54.519 510301== by 0x71B997: OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&) (OSD.cc:9701) ==00:00:02:54.519 510301== by 0x70E62E: run (OpSchedulerItem.h:148) ==00:00:02:54.519 510301== by 0x70E62E: OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (OSD.cc:10677) ==00:00:02:54.519 510301== by 0xD3C093: ShardedThreadPool::shardedthreadpool_worker(unsigned int) (WorkQueue.cc:311) ==00:00:02:54.519 510301== by 0xD3ECF3: ShardedThreadPool::WorkThreadSharded::entry() (WorkQueue.h:706) ==00:00:02:54.519 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:02:54.519 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:02:54.519 510301== Address 0xf123930 is 3,824 bytes inside a block of size 10,296 alloc'd ==00:00:02:54.519 510301== at 0xA7DC0C3: operator new[](unsigned long) (vg_replace_malloc.c:433) ==00:00:02:54.519 510301== by 0x66F766: main (ceph_osd.cc:688) ==00:00:02:54.519 510301== Block was alloc'd by thread #1 ``` Actually there is plenty of similar issues reported like: ``` ==00:00:05:04.903 510301== Possible data race during read of size 8 at 0x1E3E0588 by thread ceph#119 ==00:00:05:04.903 510301== Locks held: 1, at address 0x1EAD41D0 ==00:00:05:04.903 510301== at 0x753165: clear (hashtable.h:2051) ==00:00:05:04.903 510301== by 0x753165: std::_Hashtable<entity_addr_t, std::pair<entity_addr_t const, utime_t>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<entity_addr_t const, utime_t> >, std::__detail::_Select1st, std::equal_to<entity_addr_t>, std::hash<entity_addr_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__deta il::_Hashtable_traits<true, false, true> >::~_Hashtable() (hashtable.h:1369) ==00:00:05:04.903 510301== by 0x75331C: ~unordered_map (unordered_map.h:102) ==00:00:05:04.903 510301== by 0x75331C: OSDMap::~OSDMap() (OSDMap.h:350) ==00:00:05:04.903 510301== by 0x753606: operator() (shared_cache.hpp:100) ==00:00:05:04.903 510301== by 0x753606: std::_Sp_counted_deleter<OSDMap const*, SharedLRU<unsigned int, OSDMap const>::Cleanup, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr _base.h:471) ==00:00:05:04.903 510301== by 0x73BB26: _M_release (shared_ptr_base.h:155) ==00:00:05:04.903 510301== by 0x73BB26: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148) ==00:00:05:04.903 510301== by 0x6B58A9: ~__shared_count (shared_ptr_base.h:728) ==00:00:05:04.903 510301== by 0x6B58A9: ~__shared_ptr (shared_ptr_base.h:1167) ==00:00:05:04.903 510301== by 0x6B58A9: ~shared_ptr (shared_ptr.h:103) ==00:00:05:04.903 510301== by 0x6B58A9: OSD::create_context() (OSD.cc:9053) ==00:00:05:04.903 510301== by 0x71B570: OSD::dequeue_peering_evt(OSDShard*, PG*, std::shared_ptr<PGPeeringEvent>, ThreadPool::TPHandle&) (OSD.cc:9665) ==00:00:05:04.903 510301== by 0x71B997: OSD::dequeue_delete(OSDShard*, PG*, unsigned int, ThreadPool::TPHandle&) (OSD.cc:9701) ==00:00:05:04.903 510301== by 0x70E62E: run (OpSchedulerItem.h:148) ==00:00:05:04.903 510301== by 0x70E62E: OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (OSD.cc:10677) ==00:00:05:04.903 510301== by 0xD3C093: ShardedThreadPool::shardedthreadpool_worker(unsigned int) (WorkQueue.cc:311) ==00:00:05:04.903 510301== by 0xD3ECF3: ShardedThreadPool::WorkThreadSharded::entry() (WorkQueue.h:706) ==00:00:05:04.903 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:05:04.903 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:05:04.903 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ==00:00:05:04.903 510301== ==00:00:05:04.903 510301== This conflicts with a previous write of size 8 by thread ceph#90 ==00:00:05:04.903 510301== Locks held: 2, at addresses 0xF122A58 0xF1239A8 ==00:00:05:04.903 510301== at 0x7531E1: clear (hashtable.h:2054) ==00:00:05:04.903 510301== by 0x7531E1: std::_Hashtable<entity_addr_t, std::pair<entity_addr_t const, utime_t>, mempool::pool_allocator<(mempool::pool_index_t)15, std::pair<entity_addr_t const, utime_t> >, std::__detail::_Select1st, std::equal_to<entity_addr_t>, std::hash<entity_addr_t>, std::__detail::_Mod_range_hashing, std::__detail::_Default_ranged_hash, std::__detail::_Prime_rehash_policy, std::__detail::_Hashtable_traits<true, false, true> >::~_Hashtable() (hashtable.h:1369) ==00:00:05:04.903 510301== by 0x75331C: ~unordered_map (unordered_map.h:102) ==00:00:05:04.903 510301== by 0x75331C: OSDMap::~OSDMap() (OSDMap.h:350) ==00:00:05:04.903 510301== by 0x753606: operator() (shared_cache.hpp:100) ==00:00:05:04.903 510301== by 0x753606: std::_Sp_counted_deleter<OSDMap const*, SharedLRU<unsigned int, OSDMap const>::Cleanup, std::allocator<void>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() (shared_ptr_base.h:471) ==00:00:05:04.903 510301== by 0x73BB26: _M_release (shared_ptr_base.h:155) ==00:00:05:04.903 510301== by 0x73BB26: std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() (shared_ptr_base.h:148) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr_base.h:747) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr_base.h:1078) ==00:00:05:04.903 510301== by 0x72191E: operator= (shared_ptr.h:103) ==00:00:05:04.903 510301== by 0x72191E: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8116) ==00:00:05:04.903 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:05:04.903 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:05:04.903 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:05:04.903 510301== Address 0x1e3e0588 is 872 bytes inside a block of size 1,208 alloc'd ==00:00:05:04.903 510301== at 0xA7DC0C3: operator new[](unsigned long) (vg_replace_malloc.c:433) ==00:00:05:04.903 510301== by 0x6C7C0C: OSDService::try_get_map(unsigned int) (OSD.cc:1606) ==00:00:05:04.903 510301== by 0x7213BD: get_map (OSD.h:699) ==00:00:05:04.903 510301== by 0x7213BD: get_map (OSD.h:1732) ==00:00:05:04.903 510301== by 0x7213BD: OSD::_committed_osd_maps(unsigned int, unsigned int, MOSDMap*) (OSD.cc:8076) ==00:00:05:04.903 510301== by 0x7752CA: C_OnMapCommit::finish(int) (OSD.cc:7678) ==00:00:05:04.903 510301== by 0x72A06C: Context::complete(int) (Context.h:77) ==00:00:05:04.903 510301== by 0xD07F14: Finisher::finisher_thread_entry() (Finisher.cc:66) ==00:00:05:04.903 510301== by 0xA7E1203: mythread_wrapper (hg_intercepts.c:389) ==00:00:05:04.903 510301== by 0xC6182DD: start_thread (in /usr/lib64/libpthread-2.28.so) ==00:00:05:04.903 510301== by 0xD8B34B2: clone (in /usr/lib64/libc-2.28.so) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com> (cherry picked from commit 80da5f9) Conflicts: src/osd/OSD.cc in bool OSD::asok_command int OSD::shutdown void OSD::maybe_update_heartbeat_peers void OSD::_preboot void OSD::queue_want_up_thru void OSD::send_alive void OSD::send_failures void OSD::send_beacon MPGStats* OSD::collect_pg_stats void OSD::note_down_osd void OSD::consume_map void OSD::activate_map src/osd/OSD.h in private: dispatch_session_waiting - also use the new const OSDMapRef in places that no longer exist in master src/osd/OSD.cc in void OSDService::share_map void OSDService::send_incremental_map int OSD::_do_command void OSD::note_up_osd int OSD::init_op_flags src/osd/OSD.h in void send_incremental_map void share_map

Otherwise, if we assert, we'll hang here: Thread 1 (Thread 0x7f74eba79580 (LWP 1688617)): #0 0x00007f74eb2aa529 in futex_wait (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0x7ffd642b4b30) at pthread_cond_destroy.c:54 #3 0x0000563ff2e5a891 in LibRadosService_StatusFormat_Test::TestBody (this=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:78 #4 0x0000563ff2e9dc3a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x563ff2ea72e4 "the test body", method=<optimized out>, object=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2605 #5 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x563ff422a6d0, method=<optimized out>, location=location@entry=0x563ff2ea72e4 "the test body") at ./src/googletest/googletest/src/gtest.cc:2641 #6 0x0000563ff2e908c3 in testing::Test::Run (this=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2680 #7 0x0000563ff2e90a25 in testing::TestInfo::Run (this=0x563ff41a3b70) at ./src/googletest/googletest/src/gtest.cc:2858 #8 0x0000563ff2e90ec1 in testing::TestSuite::Run (this=0x563ff41b6230) at ./src/googletest/googletest/src/gtest.cc:3012 ceph#9 0x0000563ff2e92bdc in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at ./src/googletest/googletest/src/gtest.cc:5723 ceph#10 0x0000563ff2e9e14a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x563ff2ea8728 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x563ff41a2d10) at ./src/googletest/googletest/src/gtest.cc:2605 ceph#11 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563ff41a2d10, method=<optimized out>, location=location@entry=0x563ff2ea8728 "auxiliary test code (environments or event listeners)") at ./src/googletest/googletest/src/gtest.cc:2641 ceph#12 0x0000563ff2e90ae8 in testing::UnitTest::Run (this=0x563ff30c0660 <testing::UnitTest::GetInstance()::instance>) at ./src/googletest/googletest/src/gtest.cc:5306 Signed-off-by: Sage Weil <sage@newdream.net>

Otherwise, if we assert, we'll hang here: Thread 1 (Thread 0x7f74eba79580 (LWP 1688617)): #0 0x00007f74eb2aa529 in futex_wait (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0x7ffd642b4b30) at pthread_cond_destroy.c:54 #3 0x0000563ff2e5a891 in LibRadosService_StatusFormat_Test::TestBody (this=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:78 #4 0x0000563ff2e9dc3a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x563ff2ea72e4 "the test body", method=<optimized out>, object=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2605 #5 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x563ff422a6d0, method=<optimized out>, location=location@entry=0x563ff2ea72e4 "the test body") at ./src/googletest/googletest/src/gtest.cc:2641 #6 0x0000563ff2e908c3 in testing::Test::Run (this=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2680 #7 0x0000563ff2e90a25 in testing::TestInfo::Run (this=0x563ff41a3b70) at ./src/googletest/googletest/src/gtest.cc:2858 #8 0x0000563ff2e90ec1 in testing::TestSuite::Run (this=0x563ff41b6230) at ./src/googletest/googletest/src/gtest.cc:3012 ceph#9 0x0000563ff2e92bdc in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at ./src/googletest/googletest/src/gtest.cc:5723 ceph#10 0x0000563ff2e9e14a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x563ff2ea8728 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x563ff41a2d10) at ./src/googletest/googletest/src/gtest.cc:2605 ceph#11 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563ff41a2d10, method=<optimized out>, location=location@entry=0x563ff2ea8728 "auxiliary test code (environments or event listeners)") at ./src/googletest/googletest/src/gtest.cc:2641 ceph#12 0x0000563ff2e90ae8 in testing::UnitTest::Run (this=0x563ff30c0660 <testing::UnitTest::GetInstance()::instance>) at ./src/googletest/googletest/src/gtest.cc:5306 Signed-off-by: Sage Weil <sage@newdream.net> (cherry picked from commit ee5a0c9)

f7181ab has optimized the client parallelism. To achieve that `PG::do_osd_ops()` were converted to return basically future pair of futures. Unfortunately, the life- time management of `OpsExecuter` was kept intact. In the result, the object was valid only till fullfying the outer future while, due to the `rollbacker` instances, it should be available till `all_completed` becomes available. This issue can explain the following problem has been observed in a Teuthology job [1]. ``` DEBUG 2021-05-20 08:03:22,617 [shard 0] osd - do_op_call: method returned ret=-17, outdata.length()=0 while num_read=1, num_write=0 DEBUG 2021-05-20 08:03:22,617 [shard 0] osd - rollback_obc_if_modified: object 19:e17d4708:test-rados-api-smithi095-38404-2::foo:head got erro r generic:17, need_rollback=false ================================================================= ==33626==ERROR: AddressSanitizer: heap-use-after-free on address 0x60d0000b9320 at pc 0x560f486b8222 bp 0x7fffc467a1e0 sp 0x7fffc467a1d0 READ of size 4 at 0x60d0000b9320 thread T0 #0 0x560f486b8221 (/usr/bin/ceph-osd+0x2c610221) #1 0x560f4880c6b1 in seastar::continuation<seastar::internal::promise_base_with_type<boost::intrusive_ptr<MOSDOpReply> >, seastar::noncopy able_function<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_ wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail<crimson::osd:: IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson: :errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>, crimson::interruptible::interruptible_future_detail<crimson::osd::IOInte rruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::error ated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > >(seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail <crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_ future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>&&)::{lambda(seastar::internal::promise_base_with_type<boos t::intrusive_ptr<MOSDOpReply> >&&, seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail<crimson::osd::IOInterruptCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<(std::errc)11> > >::_future<crimson::errorated_future_marker<boost::intrusive_ptr<MOSDOpReply> > > > ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x2c7646b1) #2 0x560f5352c3ae (/usr/bin/ceph-osd+0x374843ae) #3 0x560f535318ef (/usr/bin/ceph-osd+0x374898ef) #4 0x560f536e395a (/usr/bin/ceph-osd+0x3763b95a) #5 0x560f532413d9 (/usr/bin/ceph-osd+0x371993d9) #6 0x560f476af95a in main (/usr/bin/ceph-osd+0x2b60795a) #7 0x7f7aa0af97b2 in __libc_start_main (/lib64/libc.so.6+0x237b2) #8 0x560f477d2e8d in _start (/usr/bin/ceph-osd+0x2b72ae8d) ``` [1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-05-20_07:28:16-rados-master-distro-basic-smithi/6124735/ The commit deals with the problem by repacking the outer future. An alternative could be in switching from `std::unique_ptr` to `seastar::shared_ptr` for managing `OpsExecuter`. Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

The `FuturizedStore` interface imposes the `get_attr()` takes the `name` parameter as `std::string_view`, and thus burdens implementations with extending the life- time of the data the instance refers to. Unfortunately, `AlienStore` is unaware that prolonging the life of a `std::string_view` instance doesn't prolong the data memory it points to. This problem has manifested in the following use-after-free detected at Sepia: ``` rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-05-26_12:20:26-rados-master-distro-basic-smithi/6136929$ less ./remote/smithi194/log/ceph-osd.7.log.gz ... DEBUG 2021-05-26 20:24:54,077 [shard 0] osd - do_osd_ops_execute: object 14:55e1a5b4:test-rados-api-smithi067-38889-2::foo:head - handling op call DEBUG 2021-05-26 20:24:54,077 [shard 0] osd - handling op call on object 14:55e1a5b4:test-rados-api-smithi067-38889-2::foo:head DEBUG 2021-05-26 20:24:54,078 [shard 0] osd - calling method lock.lock, num_read=0, num_write=0 DEBUG 2021-05-26 20:24:54,078 [shard 0] osd - handling op getxattr on object 14:55e1a5b4:test-rados-api-smithi067-38889-2::foo:head DEBUG 2021-05-26 20:24:54,078 [shard 0] osd - getxattr on obj=14:55e1a5b4:test-rados-api-smithi067-38889-2::foo:head for attr=_lock.TestLockPP1 DEBUG 2021-05-26 20:24:54,078 [shard 0] bluestore - get_attr ================================================================= ==34068==ERROR: AddressSanitizer: heap-use-after-free on address 0x6030001851d0 at pc 0x7f824d6a5b27 bp 0x7f822b4201c0 sp 0x7f822b41f968 READ of size 17 at 0x6030001851d0 thread T28 (alien-store-tp) ... #0 0x7f824d6a5b26 (/lib64/libasan.so.5+0x40b26) #1 0x55e2cbb2e00b (/usr/bin/ceph-osd+0x2b6dc00b) #2 0x55e2d31f086e (/usr/bin/ceph-osd+0x32d9e86e) #3 0x55e2d3467607 in crimson::os::ThreadPool::loop(std::chrono::duration<long, std::ratio<1l, 1000l> >, unsigned long) (/usr/bin/ceph-osd+0x33015607) #4 0x55e2d346b14a (/usr/bin/ceph-osd+0x3301914a) #5 0x7f8249d32ba2 (/lib64/libstdc++.so.6+0xc2ba2) #6 0x7f824a00d149 in start_thread (/lib64/libpthread.so.0+0x8149) #7 0x7f82486edf22 in clone (/lib64/libc.so.6+0xfcf22) 0x6030001851d0 is located 0 bytes inside of 31-byte region [0x6030001851d0,0x6030001851ef) freed by thread T0 here: #0 0x7f824d757688 in operator delete(void*) (/lib64/libasan.so.5+0xf2688) previously allocated by thread T0 here: #0 0x7f824d7567b0 in operator new(unsigned long) (/lib64/libasan.so.5+0xf17b0) Thread T28 (alien-store-tp) created by T0 here: #0 0x7f824d6b7ea3 in __interceptor_pthread_create (/lib64/libasan.so.5+0x52ea3) SUMMARY: AddressSanitizer: heap-use-after-free (/lib64/libasan.so.5+0x40b26) Shadow bytes around the buggy address: 0x0c06800289e0: fd fd fd fa fa fa fd fd fd fa fa fa 00 00 00 fa 0x0c06800289f0: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd 0x0c0680028a00: fd fa fa fa fd fd fd fa fa fa fd fd fd fa fa fa 0x0c0680028a10: fd fd fd fa fa fa fd fd fd fa fa fa fd fd fd fa 0x0c0680028a20: fa fa fd fd fd fa fa fa fd fd fd fa fa fa fd fd =>0x0c0680028a30: fd fd fa fa fd fd fd fd fa fa[fd]fd fd fd fa fa 0x0c0680028a40: fd fd fd fd fa fa fd fd fd fd fa fa 00 00 00 07 0x0c0680028a50: fa fa 00 00 00 fa fa fa 00 00 00 fa fa fa fd fd 0x0c0680028a60: fd fd fa fa fd fd fd fd fa fa fd fd fd fd fa fa 0x0c0680028a70: 00 00 00 00 fa fa fd fd fd fd fa fa fd fd fd fd 0x0c0680028a80: fa fa fd fd fd fd fa fa fd fd fd fd fa fa fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==34068==ABORTING ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

There is a scenario when the `active_con` is properly chosen but isn't marked as `ready_to_send`. If `renew_subs()` is called during the `on_session_opened()`, the flag will be turned on after the subscriptions are renewed which cannot happen as it requires the flag to be already set. In other words: there is a circular data dependency. The net result is stalling the subscription machinery, particularly the `OSDMap` subs. This caused a nasty peering issue at Sepia [1] where PG 2.7 got stuck in the `GetInfo` state. ``` rzarzynski@teuthology:/home/teuthworker/archive/rzarzynski-2021-05-26_12:20:26-rados-master-distro-basic-smithi/6136908$ less ./remote/smithi039/log/ceph-osd.1.log.gz ... DEBUG 2021-05-26 20:19:48,134 [shard 0] osd - pg_epoch 14 pg[2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) [] r= -1 lpr=0 crt=0'0 mlcod 0'0 unknown enter Initial ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=0 crt=0'0 mlcod 0'0 unknown enter Reset ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Start ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 unknown enter Started/Primary ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Peering ... DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetInfo DEBUG 2021-05-26 20:19:48,138 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior all_probe DEBUG 2021-05-26 20:19:48,139 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering build_prior final: probe 0,1 down blocked_by {} DEBUG 2021-05-26 20:19:48,139 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering up_thru 0 < same_since 14, must notify monitor DEBUG 2021-05-26 20:19:48,139 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: no prior_set down osds, clearing prior_readable_until_ub DEBUG 2021-05-26 20:19:48,139 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: querying info from osd.0 ... DEBUG 2021-05-26 20:19:48,237 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering got osd.0 2.7( DNE empty local-lis/les=0/0 n=0 ec=0/0 lis/c=0/0 les/c/f=0/0/0 sis=0) DEBUG 2021-05-26 20:19:48,237 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: Adding osd: 0 peer features: 3f01cfbb7ffdffff DEBUG 2021-05-26 20:19:48,237 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: Common peer features: 3f01cfbb7ffdffff DEBUG 2021-05-26 20:19:48,237 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: Common acting features: 3f01cfbb7ffdffff DEBUG 2021-05-26 20:19:48,238 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering state<Started/Primary/Peering/GetInfo>: Common upacting features: 3f01cfbb7ffdffff DEBUG 2021-05-26 20:19:48,238 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering exit Started/Primary/Peering/GetInfo 0.099480 4 2021-05-26T20:19:48.146172+0000 ... DEBUG 2021-05-26 20:19:48,238 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetLog ... DEBUG 2021-05-26 20:19:48,238 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/GetMissing ... DEBUG 2021-05-26 20:19:48,238 [shard 0] osd - pg_epoch 14 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+peering enter Started/Primary/Peering/WaitUpThru ... DEBUG 2021-05-26 20:19:49,139 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=0/0 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating enter Started/Primary/Active ... DEBUG 2021-05-26 20:19:49,142 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=0/0 les/c/f=0/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 creating+activating enter Started/Primary/Active/Activating ... DEBUG 2021-05-26 20:19:49,204 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Recovered ... DEBUG 2021-05-26 20:19:49,204 [shard 0] osd - pg_epoch 15 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Started/Primary/Active/Clean ... DEBUG 2021-05-26 20:22:31,223 [shard 0] osd - pg_epoch 86 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=14) [1,0] r=0 lpr=14 crt=0'0 mlcod 0'0 active enter Reset ... <a lot of flipping> ... DEBUG 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown activate_map DEBUG 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown exit Reset 0.035744 1 2021-05-26T20:24:07.817331+0000 INFO 2021-05-26 20:24:07,851 [shard 0] osd - Exiting state: Reset, entered at 1622060647.8158188, 1622060647.8173316 spent on 1 events DEBUG 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown enter Started INFO 2021-05-26 20:24:07,851 [shard 0] osd - Entering state: Started DEBUG 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown enter Start INFO 2021-05-26 20:24:07,851 [shard 0] osd - Entering state: Start INFO 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown state<Start>: transitioning to Primary DEBUG 2021-05-26 20:24:07,851 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown exit Start 0.000041 0 0.000000 INFO 2021-05-26 20:24:07,851 [shard 0] osd - Exiting state: Start, entered at 1622060647.8516333, 0.0 spent on 0 events DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary INFO 2021-05-26 20:24:07,852 [shard 0] osd - Entering state: Started/Primary DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163 ) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 unknown enter Started/Primary/Peering INFO 2021-05-26 20:24:07,852 [shard 0] osd - Entering state: Started/Primary/Peering DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering enter Started/Primary/Peering/GetInfo INFO 2021-05-26 20:24:07,852 [shard 0] osd - Entering state: Started/Primary/Peering/GetInfo ... DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering build_prior all_probe 0,1,4 DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering build_prior maybe_rw interval:139, acting: 0 DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering build_prior final: probe 0,1,4 down blocked_by {} DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering up_thru 125 < same_since 163, must notify monitor DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering state<Started/Primary/Peering/GetInfo>: no prior_set down osds, clearing prior_readable_until_ub DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering state<Started/Primary/Peering/GetInfo>: querying info from osd.0 DEBUG 2021-05-26 20:24:07,852 [shard 0] osd - pg_epoch 163 pg[2.7( empty local-lis/les=14/15 n=0 ec=14/14 lis/c=14/14 les/c/f=15/15/0 sis=163) [1,0] r=0 lpr=163 pi=[14,163)/1 crt=0'0 mlcod 0'0 peering state<Started/Primary/Peering/GetInfo>: querying info from osd.4 ... DEBUG 2021-05-26 20:24:07,924 [shard 0] ms - [osd.1(cluster) v2:172.21.15.39:6803/34727@61064 >> osd.4 v2:172.21.15.62:6802/34686] connect to existing DEBUG 2021-05-26 20:24:07,924 [shard 0] ms - [osd.1(cluster) v2:172.21.15.39:6803/34727@61064 >> osd.4 v2:172.21.15.62:6802/34686] --> ceph#62 === pg_query2(2.7 2.7 query(info 0'0 epoch_sent 163) e163/163) v1 (131) ... DEBUG 2021-05-26 20:24:07,942 [shard 0] ms - [osd.1(cluster) v2:172.21.15.39:6803/34727@61064 >> osd.4 v2:172.21.15.62:6802/34686] GOT AckFrame: seq=62 ... <plenty of osd_ping messanging but no reply to the pg_query for 2.7> ... DEBUG 2021-05-26 20:58:19,829 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.39:6807/34727 >> osd.4 v2:172.21.15.62:6807/34686@54816] <== ceph#772 = == osd_ping(ping e17 up_from 10 ping_stamp 2021-05-26T20:58:19.825573+0000/2319.780029297s send_stamp 2319.780029297s) v5 (70) DEBUG 2021-05-26 20:58:19,829 [shard 0] ms - [osd.1(hb_front) v2:172.21.15.39:6807/34727 >> osd.4 v2:172.21.15.62:6807/34686@54816] --> ceph#772 === osd_ping(ping_reply e249 up_from 10 ping_stamp 2021-05-26T20:58:19.825573+0000/2319.780029297s send_stamp 2320.039062500s) v5 (70 ``` The peering request got stuck due to awaiting for `OSDMap`. ``` DEBUG 2021-05-26 20:24:07,930 [shard 0] ms - [osd.4(cluster) v2:172.21.15.62:6802/34686 >> osd.1 v2:172.21.15.39:6803/34727@61064] <== ceph#62 === pg_query2(2.7 2.7 query(info 0'0 epoch_sent 163) e163/163) v1 (131) DEBUG 2021-05-26 20:24:07,930 [shard 0] osd - handle_peering_op on 2.7 from 1 DEBUG 2021-05-26 20:24:07,930 [shard 0] osd - peering_event(id=517, detail=PeeringEvent(from=1 pgid=2.7 sent=163 requested=163 evt=epoch_sent: 163 epoch_requested: 163 MQuery 2.7 from 1 query_epoch 163 query: query(info 0'0 epoch_sent 163))): star ``` ``` INFO 2021-05-26 20:19:49,127 [shard 0] osd - evt epoch is 15, i have 14, will wait INFO 2021-05-26 20:19:49,128 [shard 0] osd - osdmap_subscribe(14) DEBUG 2021-05-26 20:19:49,128 [shard 0] ms - [osd.4(client) v2:172.21.15.62:6801/34686@63208 >> mon.1 v2:172.21.15.62:3300/0] --> ceph#9 === mon_s ubscribe({osdmap=14}) v3 (15) ... INFO 2021-05-26 20:19:49,131 [shard 0] osd - handle_osd_map osd_map(14..15 src has 1..15) v4 INFO 2021-05-26 20:19:49,131 [shard 0] osd - handle_osd_map epochs [14..15], i have 15, src has [1..15] ... INFO 2021-05-26 20:19:49,138 [shard 0] osd - handle_osd_map osd_map(14..15 src has 1..15) v4 INFO 2021-05-26 20:19:49,138 [shard 0] osd - handle_osd_map epochs [14..15], i have 15, src has [1..15] ... INFO 2021-05-26 20:19:49,139 [shard 0] osd - evt epoch is 15, i have 14, will wait INFO 2021-05-26 20:19:49,141 [shard 0] osd - osdmap_subscribe(14) WARN 2021-05-26 20:19:49,141 [shard 0] monc - renew_subs - empty ... INFO 2021-05-26 20:19:50,140 [shard 0] osd - handle_osd_map osd_map(15..16 src has 1..16) v4 INFO 2021-05-26 20:19:50,140 [shard 0] osd - handle_osd_map epochs [15..16], i have 15, src has [1..16] DEBUG 2021-05-26 20:19:50,141 [shard 0] bluestore - do_transaction INFO 2021-05-26 20:19:50,145 [shard 0] osd - osd.4: committed_osd_maps(16, 16) ... INFO 2021-05-26 20:20:42,881 [shard 0] osd - handle_osd_map epochs [16..17], i have 16, src has [1..17] DEBUG 2021-05-26 20:20:42,882 [shard 0] bluestore - do_transaction INFO 2021-05-26 20:20:42,886 [shard 0] osd - osd.4: committed_osd_maps(17, 17) ... INFO 2021-05-26 20:20:43,941 [shard 0] osd - evt epoch is 18, i have 17, will wait INFO 2021-05-26 20:20:43,941 [shard 0] osd - osdmap_subscribe(17) ... INFO 2021-05-26 20:20:43,957 [shard 0] osd - evt epoch is 18, i have 17, will wait INFO 2021-05-26 20:20:43,957 [shard 0] osd - osdmap_subscribe(17) ... INFO 2021-05-26 20:20:43,969 [shard 0] osd - evt epoch is 18, i have 17, will wait INFO 2021-05-26 20:20:43,969 [shard 0] osd - osdmap_subscribe(17) ... DEBUG 2021-05-26 20:20:46,930 [shard 0] ms - [osd.4(client) v2:172.21.15.62:6801/34686@57288 >> mon.2 v2:172.21.15.39:3301/0] <== #4 === osd_m ap(20..21 src has 1..21) v4 (41) INFO 2021-05-26 20:20:46,930 [shard 0] osd - handle_osd_map osd_map(20..21 src has 1..21) v4 INFO 2021-05-26 20:20:46,930 [shard 0] osd - handle_osd_map epochs [20..21], i have 17, src has [1..21] INFO 2021-05-26 20:20:46,930 [shard 0] osd - handle_osd_map message skips epochs 18..19 INFO 2021-05-26 20:20:46,930 [shard 0] osd - osdmap_subscribe(18) ... DEBUG 2021-05-26 20:20:47,936 [shard 0] ms - [osd.4(client) v2:172.21.15.62:6801/34686@57288 >> mon.2 v2:172.21.15.39:3301/0] <== #5 === osd_m ap(21..22 src has 1..22) v4 (41) INFO 2021-05-26 20:20:47,936 [shard 0] osd - handle_osd_map osd_map(21..22 src has 1..22) v4 INFO 2021-05-26 20:20:47,936 [shard 0] osd - handle_osd_map epochs [21..22], i have 17, src has [1..22] INFO 2021-05-26 20:20:47,936 [shard 0] osd - handle_osd_map message skips epochs 18..20 INFO 2021-05-26 20:20:47,936 [shard 0] osd - osdmap_subscribe(18) ... <osdmap_subscribe(18) over and over> ``` ``` 2021-05-26T20:19:42.048+0000 7f4712ffd700 1 -- [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] <== osd.4 v2:172.21.15.62:6801/34686 4 ==== mon_subscribe({mgrmap=0+,osd_pg_creates=0+,osdmap=0+}) v3 ==== 82+0+0 (secure 0 0 0) 0x7f46fc04e150 con 0x7f470401c480 2021-05-26T20:19:42.048+0000 7f4712ffd700 20 mon.b@1(peon) e1 _ms_dispatch existing session 0x7f46fc02f500 for osd.4 2021-05-26T20:19:42.048+0000 7f4712ffd700 20 mon.b@1(peon) e1 entity_name osd.4 global_id 4168 (new_ok) caps allow * 2021-05-26T20:19:42.048+0000 7f4712ffd700 10 mon.b@1(peon) e1 handle_subscribe mon_subscribe({mgrmap=0+,osd_pg_creates=0+,osdmap=0+}) v3 ... 2021-05-26T20:19:49.129+0000 7f4712ffd700 1 -- [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] <== osd.4 v2:172.21.15.62:6801/34686 9 ==== mo n_subscribe({osdmap=14}) v3 ==== 36+0+0 (secure 0 0 0) 0x7f46e8556210 con 0x7f470401c480 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 mon.b@1(peon) e1 _ms_dispatch existing session 0x7f46fc02f500 for osd.4 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 mon.b@1(peon) e1 entity_name osd.4 global_id 4168 (new_ok) caps allow * 2021-05-26T20:19:49.129+0000 7f4712ffd700 10 mon.b@1(peon) e1 handle_subscribe mon_subscribe({osdmap=14}) v3 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 is_capable service=mon command= read addr v2:172.21.15.62:6801/34686 on cap allow * 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 allow so far , doing grant allow * 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 allow all 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 is_capable service=osd command= read addr v2:172.21.15.62:6801/34686 on cap allow * 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 allow so far , doing grant allow * 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 allow all 2021-05-26T20:19:49.129+0000 7f4712ffd700 10 mon.b@1(peon).osd e15 check_osdmap_sub 0x7f46e84f0150 next 14 (onetime) 2021-05-26T20:19:49.129+0000 7f4712ffd700 5 mon.b@1(peon).osd e15 send_incremental [14..15] to osd.4 2021-05-26T20:19:49.129+0000 7f4712ffd700 10 mon.b@1(peon).osd e15 build_incremental [14..15] with features 3f01cfbb7ffdffff 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 mon.b@1(peon).osd e15 build_incremental inc 15 622 bytes 2021-05-26T20:19:49.129+0000 7f4712ffd700 20 mon.b@1(peon).osd e15 build_incremental inc 14 578 bytes 2021-05-26T20:19:49.129+0000 7f4712ffd700 1 -- [v2:172.21.15.62:3300/0,v1:172.21.15.62:6789/0] --> v2:172.21.15.62:6801/34686 -- osd_map(14.. 15 src has 1..15) v4 -- 0x7f46e856a100 con 0x7f470401c480 ``` ``` seastar::future<> Client::renew_subs() { if (!sub.have_new()) { logger().warn("{} - empty", __func__); return seastar::now(); } logger().trace("{}", __func__); auto m = crimson::make_message<MMonSubscribe>(); m->what = sub.get_subs(); m->hostname = ceph_get_short_hostname(); return send_message(std::move(m)).then([this] { sub.renewed(); }); } ``` ``` INFO 2021-05-26 20:19:42,081 [shard 0] osd - osdmap_subscribe(1) DEBUG 2021-05-26 20:19:42,081 [shard 0] ms - [osd.4(client) v2:172.21.15.62:6801/34686@63208 >> mon.1 v2:172.21.15.62:3300/0] --> #6 === mon_s ubscribe({osdmap=1}) v3 (15) ... INFO 2021-05-26 20:19:49,128 [shard 0] osd - osdmap_subscribe(14) DEBUG 2021-05-26 20:19:49,128 [shard 0] ms - [osd.4(client) v2:172.21.15.62:6801/34686@63208 >> mon.1 v2:172.21.15.62:3300/0] --> ceph#9 === mon_subscribe({osdmap=14}) v3 (15) ... INFO 2021-05-26 20:19:49,141 [shard 0] osd - osdmap_subscribe(14) WARN 2021-05-26 20:19:49,141 [shard 0] monc - renew_subs - empty <no MMonSubcribe> ... INFO 2021-05-26 20:20:43,941 [shard 0] osd - evt epoch is 18, i have 17, will wait INFO 2021-05-26 20:20:43,941 [shard 0] osd - osdmap_subscribe(17) <no MMonSubcribe> ... INFO 2021-05-26 20:20:46,930 [shard 0] osd - handle_osd_map message skips epochs 18..19 INFO 2021-05-26 20:20:46,930 [shard 0] osd - osdmap_subscribe(18) <no MMonSubcribe> ``` [1]: http://pulpito.front.sepia.ceph.com/rzarzynski-2021-05-26_12:20:26-rados-master-distro-basic-smithi/6136908 Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

exit() will call pthread_cond_destroy attempting to destroy dpdk::eal::cond upon which other threads are currently blocked results in undefine behavior. Link different libc version test, libc-2.17 can exit, libc-2.27 will deadlock, the call stack is as follows: Thread 3 (Thread 0xffff7e5749f0 (LWP 62213)): #0 0x0000ffff7f3c422c in futex_wait_cancelable (private=<optimized out>, expected=0, futex_word=0xaaaadc0e30f4 <dpdk::eal::cond+44>) at ../sysdeps/unix/sysv/linux/futex-internal.h:88 #1 __pthread_cond_wait_common (abstime=0x0, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>, cond=0xaaaadc0e30c8 <dpdk::eal::cond>) at pthread_cond_wait.c:502 #2 __pthread_cond_wait (cond=0xaaaadc0e30c8 <dpdk::eal::cond>, mutex=0xaaaadc0e30f8 <dpdk::eal::lock>) at pthread_cond_wait.c:655 #3 0x0000ffff7f1f1f80 in std::condition_variable::wait(std::unique_lock<std::mutex>&) () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6 #4 0x0000aaaad37f5078 in dpdk::eal::<lambda()>::operator()(void) const (__closure=<optimized out>, __closure=<optimized out>) at ./src/msg/async/dpdk/dpdk_rte.cc:136 #5 0x0000ffff7f1f7ed4 in ?? () from /usr/lib/aarch64-linux-gnu/libstdc++.so.6 #6 0x0000ffff7f3be088 in start_thread (arg=0xffffe73e197f) at pthread_create.c:463 #7 0x0000ffff7efc74ec in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78 Thread 1 (Thread 0xffff7ee3b010 (LWP 62200)): #0 0x0000ffff7f3c3c38 in futex_wait (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=12, futex_word=0xaaaadc0e30ec <dpdk::eal::cond+36>) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0xaaaadc0e30c8 <dpdk::eal::cond>) at pthread_cond_destroy.c:54 #3 0x0000ffff7ef2be34 in __run_exit_handlers (status=-6, listp=0xffff7f04a5a0 <__exit_funcs>, run_list_atexit=255, run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at exit.c:108 #4 0x0000ffff7ef2bf6c in __GI_exit (status=<optimized out>) at exit.c:139 #5 0x0000ffff7ef176e4 in __libc_start_main (main=0x0, argc=0, argv=0x0, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=<optimized out>) at ../csu/libc-start.c:344 #6 0x0000aaaad2939db0 in _start () at ./src/include/buffer.h:642 Fixes: https://tracker.ceph.com/issues/42890 Signed-off-by: Chunsong Feng <fengchunsong@huawei.com> Signed-off-by: luo rixin <luorixin@huawei.com>

The problem is: ``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) #1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) #2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 #3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 #4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 #5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 #6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 #7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 #8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) #1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 #2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Before the patch there was a possibility that `OSDConnectionPriv` gets destructed before a `PipelineHandle` instance that was using it. The reason is our remote-handling operations store `conn` directly while `handle` is defined in a parent class. Due to the language rules the former gets deinitialized earlier. ``` ==756032==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000039684 at pc 0x0000020bdfa2 bp 0x7ffd3abfa370 sp 0x7ffd3abfa360 READ of size 1 at 0x615000039684 thread T0 Reactor stalled for 261 ms on shard 0. Backtrace: 0x45d9d 0xe90f6d1 0xe6b8a1d 0xe6d1205 0xe6d16a8 0xe6d1938 0xe6d1c03 0x12cdf 0xccebf 0x7f6447161b1e 0x7f644714aee8 0x7f644714eed6 0x7f644714fb36 0x7f64471420b5 0x 7f6447143f3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0xbdc1a 0x20bdfa1 0x20c184e 0x352eb7f 0x352fa28 0x20b04a5 0x1be30e5 0xe694bc4 0xe6ebb8a 0xe843a11 0xe845a22 0xe29f497 0xe2a3ccd 0x1ab1841 0x3aca2 0x175698d #0 0x20bdfa1 in seastar::shared_mutex::unlock() ../src/seastar/include/seastar/core/shared_mutex.hh:122 #1 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::exit() ../src/crimson/common/operation.h:548 #2 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::exit() ../src/crimson/common/operation.h:533 #3 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::cancel() ../src/crimson/common/operation.h:539 #4 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:543 #5 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:544 #6 0x352eb7f in std::default_delete<crimson::PipelineExitBarrierI>::operator()(crimson::PipelineExitBarrierI*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85 #7 0x352eb7f in std::unique_ptr<crimson::PipelineExitBarrierI, std::default_delete<crimson::PipelineExitBarrierI> >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361 #8 0x352eb7f in crimson::PipelineHandle::~PipelineHandle() ../src/crimson/common/operation.h:457 ceph#9 0x352eb7f in crimson::osd::PhasedOperationT<crimson::osd::ClientRequest>::~PhasedOperationT() ../src/crimson/osd/osd_operation.h:152 ceph#10 0x352eb7f in crimson::osd::ClientRequest::~ClientRequest() ../src/crimson/osd/osd_operations/client_request.cc:64 ceph#11 ... ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

before this change, we allocate memory chunks with specified size using `new []`, but we never free them. when testing with LeakSanitizer enabled, it rightly points identifies the leakage: ``` Direct leak of 8754 byte(s) in 184 object(s) allocated from: #0 0x55c0b2470f0d in operator new[](unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_memory+0x196f0d) (BuildId: d3267dd8819427b804c4729e0467dbe7601fb321) #1 0x55c0b247456c in MemoryIsZeroSmallTest_MemoryIsZeroTestSmall_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/common/test_memory.cc:33:18 #2 0x55c0b2598ee6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 #3 0x55c0b2553b92 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 #4 0x55c0b25049dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 #5 0x55c0b2506a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11 #6 0x55c0b250804b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28 #7 0x55c0b25254d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44 #8 0x55c0b25a16f6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#9 0x55c0b255a502 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#10 0x55c0b2524862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10 ceph#11 0x55c0b24ab4c0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#12 0x55c0b24ab451 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#13 0x7f45e065ad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` in this change, we free the allocate memory. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

before this change, we increment the refcount when constructing `cct` instrusive_ptr, but nobody owns this smart pointer. also, `CephContext` 's constructor set its refcount to 1. so, when the test finishes, the refcount is 1, and this leads to a leakage of the `CephContext` instance, this not only annoys ASan, and defeats the purpose of 14d878c. ``` Indirect leak of 10880000 byte(s) in 1 object(s) allocated from: #0 0x5564d173537d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_ipaddr+0x19b37d) (BuildId: 45c0c7f28b253c04fcb7bb1a43aed52a5526d734) #1 0x7fe7f2ccd189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x7fe7f2ccc563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x7fe7f2ccc563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39 #4 0x7fe7f2ccc2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18 #5 0x7fe7f2cc6192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9 #6 0x7fe7f2cb91e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5 #7 0x7fe7f1f8f96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16 #8 0x7fe7f1f8e93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5 ceph#9 0x5564d1752eb9 in pick_address_find_ip_in_subnet_list_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/test_ipaddr.cc:706:47 ceph#10 0x5564d18694d6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#11 0x5564d1820fc2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#12 0x5564d17d19dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 ceph#13 0x5564d17d3a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11 ceph#14 0x5564d17d504b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28 ceph#15 0x5564d17f24d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44 ceph#16 0x5564d1871d06 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#17 0x5564d1827932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#18 0x5564d17f1862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10 ceph#19 0x5564d1775d80 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#20 0x5564d1775d11 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10 ``` so, in this change, we do not increase the refcount when creating cct. the same applies to `test/common/test_fault_injector.cc`. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

before this change, in test_util.cc, we increment the refcount of when constructing it. but at that moment, nobody really owns it. also, `CephContext` 's refcount is set to 1 in its constructor. so, we should not do this. otherwise, the created `CephContext` is leaked as LeakSanitizer rightly points out: ``` Indirect leak of 10880000 byte(s) in 1 object(s) allocated from: #0 0x5632320d27ed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_util+0x1917ed) (BuildId: ff1df1455bd07b651ad580584a17ea204afeb36e) #1 0x7ff9d535b189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x7ff9d535a563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x7ff9d535a563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39 #4 0x7ff9d535a2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18 #5 0x7ff9d5354192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9 #6 0x7ff9d53471e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5 #7 0x7ff9d461d96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16 #8 0x7ff9d461c93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5 ceph#9 0x5632320d52e0 in util_collect_sys_info_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/common/test_util.cc:34:27 ceph#10 0x563232205c16 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#11 0x5632321c2742 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#12 0x5632321736dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 ``` in this change, instead of using a raw pointer, let's use `boost::intrusive_ptr<CephContext>` to manage the lifecyle of `CephContext`, this also address the leakage reported by LeakSanitizer. the same applies to common/test_context.cc Signed-off-by: Kefu Chai <tchaikov@gmail.com>

…sive_ptr<CephContext> before this change, we increment the refcount when constructing `cct` instrusive_ptr, but nobody owns this smart pointer. also, `CephContext` 's constructor set its refcount to 1. so, when the test finishes, the refcount is 1, and this leads to a leakage of the `CephContext` instance. and LeakSanitizer points this out: ``` Indirect leak of 10880000 byte(s) in 1 object(s) allocated from: #0 0xaaaac359c7c8 in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/bin/unittest_rgw_iam_policy+0x211c7c8) (BuildId: 060fadb10da261b52fd5757c7b1e9812d34542f1) #1 0xffff96f764e4 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0xffff96f757cc in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0xffff96f757cc in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:2396:39 #4 0xffff96f75500 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:2494:18 #5 0xffff96f6ec4c in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:1039:9 #6 0xffff96f63528 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/log/Log.cc:53:5 #7 0xffff96045300 in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/common/ceph_context.cc:729:16 #8 0xffff960446ec in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/common/ceph_context.cc:697:5 ceph#9 0xaaaac3629238 in IPPolicyTest::IPPolicyTest() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/rgw/test_rgw_iam_policy.cc:864:15 ceph#10 0xaaaac3628da0 in IPPolicyTest_MaskedIPOperations_Test::IPPolicyTest_MaskedIPOperations_Test() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/rgw/test_rgw_iam_policy.cc:869:1 ceph#11 0xaaaac3628d3c in testing::internal::TestFactoryImpl<IPPolicyTest_MaskedIPOperations_Test>::CreateTest() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/googletest/googletest/include/gtest/internal/gtest-internal.h:472:44 ``` so, in this change, we do not increase the refcount when creating cct. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

before this change, we create a new cct instance with `new`, but we never free this instance after done with it. and LeakSanitizer points this out: ``` Indirect leak of 10880000 byte(s) in 1 object(s) allocated from: #0 0x561afe148fed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_config_map+0x1c2fed) (BuildId: 3ce9eeed38cee335628fa74fdd08cd215b15019e) #1 0x7f37dc9ac189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x7f37dc9ab563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x7f37dc9ab563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39 #4 0x7f37dc9ab2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18 #5 0x7f37dc9a5192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9 #6 0x7f37dc9981e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5 #7 0x7f37dbc6e96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16 #8 0x7f37dbc6d93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5 ceph#9 0x561afe14e983 in ConfigMap_add_option_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_config_map.cc:58:18 ceph#10 0x561afe2689b6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#11 0x561afe221262 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#12 0x561afe1d1f7c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 ceph#13 0x561afe1d3fb2 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11 ceph#14 0x561afe1d55eb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28 ceph#15 0x561afe1f2a78 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44 ceph#16 0x561afe2711e6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#17 0x561afe227bd2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#18 0x561afe1f1e02 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10 ceph#19 0x561afe176ec0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#20 0x561afe176e51 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#21 0x7f37d9397d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` so in this change, we manage the `CephContext` pointer with a smart pointer. because the size of CephContext could be large, we don't create it on stack. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

before this change, we create a new CrushWrapper instance with `new`, but we never free this instance after done with it. and LeakSanitizer points this out: ``` Direct leak of 544 byte(s) in 1 object(s) allocated from: #0 0x561afe148fed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_config_map+0x1c2fed) (BuildId: 3ce9eeed38cee335628fa74fdd08cd215b15019e) #1 0x561afe151cbd in ConfigMap_result_sections_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_config_map.cc:93:16 #2 0x561afe2689b6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 #3 0x561afe221262 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 #4 0x561afe1d1f7c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 #5 0x561afe1d3fb2 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11 #6 0x561afe1d55eb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28 #7 0x561afe1f2a78 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44 #8 0x561afe2711e6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 ceph#9 0x561afe227bd2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 ceph#10 0x561afe1f1e02 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10 ceph#11 0x561afe176ec0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#12 0x561afe176e51 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#13 0x7f37d9397d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` so in this change, we manage the `CrushWrapper` pointer with a smart pointer. because the size of `CrushWrapper` is relatively large, we don't create it on stack. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

… overflow() When sanitizer is enabled, unittest_log fails as following ``` [ RUN ] Log.StderrPipeBig ================================================================= ==3302372==ERROR: AddressSanitizer: heap-use-after-free on address 0xffff96e01d00 at pc 0xaaaadd3db754 bp 0xffffd9ebffa0 sp 0xffffd9ebf790 READ of size 4096 at 0xffff96e01d00 thread T0 #0 0xaaaadd3db750 in __asan_memmove (/root/ceph-19.0.0/build/bin/unittest_log+0x3fb750) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) #1 0xffffafc23734 in char const* boost::container::dtl::memmove_n_source<char const*, char*>(char const*, unsigned long, char*) /root/ceph-19.0.0/build/boost/include/boost/container/detail/copy_move_algo.hpp:261:10 #2 0xffffafc23734 in boost::container::dtl::enable_if_memtransfer_copy_constructible<char const*, char*, char const*>::type boost::container::uninitialized_copy_alloc_n_source<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*, char*>(boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>&, char const*, unsigned long, char*) /root/ceph-19.0.0/build/boost/include/boost/container/detail/copy_move_algo.hpp:600:11 #3 0xffffafc23734 in void boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*>::uninitialized_copy_n_and_update<char*>(boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>&, char*, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/detail/advanced_insert_int.hpp:85:22 #4 0xffffafc23734 in void boost::container::expand_forward_and_insert_alloc<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char*, boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*> >(boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>&, char*, char*, unsigned long, boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*>) /root/ceph-19.0.0/build/boost/include/boost/container/detail/copy_move_algo.hpp:1469:23 #5 0xffffafc23734 in void boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_insert_forward_range_expand_forward<boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*> >(char*, unsigned long, boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*>, boost::move_detail::integral_constant<bool, false>) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:3058:7 #6 0xffffafc23734 in boost::container::vec_iterator<char*, false> boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_insert_forward_range<boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*> >(char* const&, unsigned long, boost::container::dtl::insert_range_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const*>) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2890:16 #7 0xffffafc23734 in boost::container::vec_iterator<char*, false> boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::insert<char const*>(boost::container::vec_iterator<char*, true>, char const*, char const*, boost::move_detail::disable_if_or<void, boost::move_detail::is_convertible<char const*, unsigned long>, boost::container::dtl::is_input_iterator<char const*, has_iterator_category<char const*>::value>, boost::move_detail::bool_<false>, boost::move_detail::bool_<false> >::type*) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2088:20 #8 0xffffafc23734 in ceph::logging::ConcreteEntry::ConcreteEntry(ceph::logging::Entry const&) /root/ceph-19.0.0/src/log/Entry.h:84:9 ceph#9 0xffffafc21a88 in decltype(new ((void*)(0))ceph::logging::ConcreteEntry(std::declval<ceph::logging::Entry>())) std::construct_at<ceph::logging::ConcreteEntry, ceph::logging::Entry>(ceph::logging::ConcreteEntry*, ceph::logging::Entry&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 ceph#10 0xffffafc21198 in void std::allocator_traits<std::allocator<ceph::logging::ConcreteEntry> >::construct<ceph::logging::ConcreteEntry, ceph::logging::Entry>(std::allocator<ceph::logging::ConcreteEntry>&, ceph::logging::ConcreteEntry*, ceph::logging::Entry&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:518:4 ceph#11 0xffffafc16464 in ceph::logging::ConcreteEntry& std::vector<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::emplace_back<ceph::logging::Entry>(ceph::logging::Entry&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:115:6 ceph#12 0xffffafc0dcbc in ceph::logging::Log::submit_entry(ceph::logging::Entry&&) /root/ceph-19.0.0/src/log/Log.cc:265:9 ceph#13 0xaaaadd41a404 in Log_StderrPipeBig_Test::TestBody() /root/ceph-19.0.0/src/log/test.cc:280:9 ceph#14 0xaaaade0b4338 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#15 0xaaaade061244 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#16 0xaaaade012680 in testing::Test::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2680:5 ceph#17 0xaaaade0145c4 in testing::TestInfo::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2858:11 ceph#18 0xaaaade015bc4 in testing::TestSuite::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:3012:28 ceph#19 0xaaaade031988 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5723:44 ceph#20 0xaaaade0be24c in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#21 0xaaaade0687dc in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#22 0xaaaade030e00 in testing::UnitTest::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5306:10 ceph#23 0xaaaadd425c48 in RUN_ALL_TESTS() /root/ceph-19.0.0/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#24 0xaaaadd4207a0 in main /root/ceph-19.0.0/src/log/test.cc:503:10 ceph#25 0xffffac3473f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#26 0xffffac3474c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#27 0xaaaadd364d6c in _start (/root/ceph-19.0.0/build/bin/unittest_log+0x384d6c) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) 0xffff96e01d00 is located 0 bytes inside of 6553-byte region [0xffff96e01d00,0xffff96e03699) freed by thread T0 here: #0 0xaaaadd4136f0 in operator delete(void*) (/root/ceph-19.0.0/build/bin/unittest_log+0x4336f0) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) #1 0xaaaadd434968 in boost::container::new_allocator<char>::deallocate(char*, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/new_allocator.hpp:171:7 #2 0xaaaadd434934 in boost::container::allocator_traits<boost::container::new_allocator<char> >::deallocate(boost::container::new_allocator<char>&, char*, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/allocator_traits.hpp:308:9 #3 0xaaaadd434934 in boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>::deallocate(char*, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/small_vector.hpp:255:10 #4 0xaaaadd43911c in boost::container::allocator_traits<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void> >::deallocate(boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>&, char*, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/allocator_traits.hpp:308:9 #5 0xaaaadd43911c in boost::container::vector_alloc_holder<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, unsigned long, boost::move_detail::integral_constant<unsigned int, 1u> >::deallocate(char* const&, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:487:7 #6 0xaaaadd43911c in void boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_insert_forward_range_new_allocation<boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&> >(char*, unsigned long, char*, unsigned long, boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&>) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:3080:25 #7 0xaaaadd438aec in boost::container::vec_iterator<char*, false> boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_insert_forward_range_no_capacity<boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&> >(char*, unsigned long, boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&>, boost::move_detail::integral_constant<unsigned int, 1u>) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2830:13 #8 0xaaaadd4328bc in char& boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::emplace_back<char const&>(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:1888:24 ceph#9 0xaaaadd4328bc in void boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_push_back<char const&>(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2746:13 ceph#10 0xaaaadd4328bc in boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::push_back(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:1996:4 ceph#11 0xaaaadd4328bc in StackStringBuf<4096ul>::overflow(int) /root/ceph-19.0.0/src/common/StackStringStream.h:79:11 ceph#12 0xffffac6d3dac in std::ostream::put(char) (/lib/aarch64-linux-gnu/libstdc++.so.6+0x133dac) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#13 0xffffac6d4aac in std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char) (/lib/aarch64-linux-gnu/libstdc++.so.6+0x134aac) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#14 0xaaaadd41a3c8 in Log_StderrPipeBig_Test::TestBody() /root/ceph-19.0.0/src/log/test.cc:278:9 ceph#15 0xaaaade0b4338 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#16 0xaaaade061244 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#17 0xaaaade012680 in testing::Test::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2680:5 ceph#18 0xaaaade0145c4 in testing::TestInfo::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2858:11 ceph#19 0xaaaade015bc4 in testing::TestSuite::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:3012:28 ceph#20 0xaaaade031988 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5723:44 ceph#21 0xaaaade0be24c in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#22 0xaaaade0687dc in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#23 0xaaaade030e00 in testing::UnitTest::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5306:10 ceph#24 0xaaaadd425c48 in RUN_ALL_TESTS() /root/ceph-19.0.0/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#25 0xaaaadd4207a0 in main /root/ceph-19.0.0/src/log/test.cc:503:10 ceph#26 0xffffac3473f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#27 0xffffac3474c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#28 0xaaaadd364d6c in _start (/root/ceph-19.0.0/build/bin/unittest_log+0x384d6c) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) previously allocated by thread T0 here: #0 0xaaaadd412e88 in operator new(unsigned long) (/root/ceph-19.0.0/build/bin/unittest_log+0x432e88) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) #1 0xaaaadd433ec0 in boost::container::new_allocator<char>::allocate(unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/new_allocator.hpp:160:30 #2 0xaaaadd438a68 in boost::container::allocator_traits<boost::container::new_allocator<char> >::priv_allocate(boost::move_detail::integral_constant<bool, false>, boost::container::new_allocator<char>&, unsigned long, void const*) /root/ceph-19.0.0/build/boost/include/boost/container/allocator_traits.hpp:395:16 #3 0xaaaadd438a68 in boost::container::allocator_traits<boost::container::new_allocator<char> >::allocate(boost::container::new_allocator<char>&, unsigned long, void const*) /root/ceph-19.0.0/build/boost/include/boost/container/allocator_traits.hpp:318:14 #4 0xaaaadd438a68 in boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>::allocate(unsigned long, void const*) /root/ceph-19.0.0/build/boost/include/boost/container/small_vector.hpp:248:14 #5 0xaaaadd438a68 in boost::container::allocator_traits<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void> >::allocate(boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>&, unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/allocator_traits.hpp:302:16 #6 0xaaaadd438a68 in boost::container::vector_alloc_holder<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, unsigned long, boost::move_detail::integral_constant<unsigned int, 1u> >::allocate(unsigned long) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:482:14 #7 0xaaaadd438a68 in boost::container::vec_iterator<char*, false> boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_insert_forward_range_no_capacity<boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&> >(char*, unsigned long, boost::container::dtl::insert_emplace_proxy<boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, char const&>, boost::move_detail::integral_constant<unsigned int, 1u>) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2826:73 #8 0xaaaadd4328bc in char& boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::emplace_back<char const&>(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:1888:24 ceph#9 0xaaaadd4328bc in void boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::priv_push_back<char const&>(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:2746:13 ceph#10 0xaaaadd4328bc in boost::container::vector<char, boost::container::small_vector_allocator<char, boost::container::new_allocator<void>, void>, void>::push_back(char const&) /root/ceph-19.0.0/build/boost/include/boost/container/vector.hpp:1996:4 ceph#11 0xaaaadd4328bc in StackStringBuf<4096ul>::overflow(int) /root/ceph-19.0.0/src/common/StackStringStream.h:79:11 ceph#12 0xffffac6d3dac in std::ostream::put(char) (/lib/aarch64-linux-gnu/libstdc++.so.6+0x133dac) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#13 0xffffac6d4aac in std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char) (/lib/aarch64-linux-gnu/libstdc++.so.6+0x134aac) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#14 0xaaaadd41a3c8 in Log_StderrPipeBig_Test::TestBody() /root/ceph-19.0.0/src/log/test.cc:278:9 ceph#15 0xaaaade0b4338 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#16 0xaaaade061244 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#17 0xaaaade012680 in testing::Test::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2680:5 ceph#18 0xaaaade0145c4 in testing::TestInfo::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2858:11 ceph#19 0xaaaade015bc4 in testing::TestSuite::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:3012:28 ceph#20 0xaaaade031988 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5723:44 ceph#21 0xaaaade0be24c in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2605:10 ceph#22 0xaaaade0687dc in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:2641:14 ceph#23 0xaaaade030e00 in testing::UnitTest::Run() /root/ceph-19.0.0/src/googletest/googletest/src/gtest.cc:5306:10 ceph#24 0xaaaadd425c48 in RUN_ALL_TESTS() /root/ceph-19.0.0/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#25 0xaaaadd4207a0 in main /root/ceph-19.0.0/src/log/test.cc:503:10 ceph#26 0xffffac3473f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#27 0xffffac3474c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#28 0xaaaadd364d6c in _start (/root/ceph-19.0.0/build/bin/unittest_log+0x384d6c) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) SUMMARY: AddressSanitizer: heap-use-after-free (/root/ceph-19.0.0/build/bin/unittest_log+0x3fb750) (BuildId: 6fd965435d12fd345de38dddc8723053b9877409) in __asan_memmove Shadow bytes around the buggy address: 0x200ff2dc0350: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x200ff2dc0360: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x200ff2dc0370: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x200ff2dc0380: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa 0x200ff2dc0390: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa =>0x200ff2dc03a0:[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x200ff2dc03b0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x200ff2dc03c0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x200ff2dc03d0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x200ff2dc03e0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd 0x200ff2dc03f0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd Shadow byte legend (one shadow byte represents 8 application bytes): Addressable: 00 Partially addressable: 01 02 03 04 05 06 07 Heap left redzone: fa Freed heap region: fd Stack left redzone: f1 Stack mid redzone: f2 Stack right redzone: f3 Stack after return: f5 Stack use after scope: f8 Global redzone: f9 Global init order: f6 Poisoned by user: f7 Container overflow: fc Array cookie: ac Intra object redzone: bb ASan internal: fe Left alloca redzone: ca Right alloca redzone: cb ==3302372==ABORTING ``` vec.push_back(str) will allocate memory and release the old one once there is insufficient memory which causing the old one to be invalid. So streambuf's data pointer and insertion position should be updated to newly allocated memory's address in vec. Fixes: https://tracker.ceph.com/issues/65805 Signed-off-by: Rongqi Sun <sunrongqi@huawei.com> (cherry picked from commit c8d51b9)

in this test, if the connection is blocked, the allocated `ConnectionTracker` is leaked. as pointed out by ASan: ``` Indirect leak of 506880 byte(s) in 10560 object(s) allocated from: #0 0x563e9d9ea1ed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_mon_election+0x2021ed) (BuildId: 6a9fb1b76c5d1db8d2bc9957316994f90b45b6c8) #1 0x563e9da588a6 in __gnu_cxx::new_allocator<std::_Rb_tree_node<std::pair<int const, double> > >::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x563e9da58830 in std::allocator<std::_Rb_tree_node<std::pair<int const, double> > >::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x563e9da58830 in std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<int const, double> > > >::allocate(std::allocator<std::_Rb_tree_node<std::pair<int const, double> > >&, unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20 #4 0x563e9da58701 in std::_Rb_tree<int, std::pair<int const, double>, std::_Select1st<std::pair<int const, double> >, std::less<int>, std::allocator<std::pair<int const, double> > >::_M_get_node() /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:561:16 #5 0x563e9db6f424 in std::_Rb_tree_node<std::pair<int const, double> >* std::_Rb_tree<int, std::pair<int const, double>, std::_Select1st<std::pair<int const, double> >, std::less<int>, std::allocator<std::pair<int const, double> > >::_M_create_node<std::piecewise_construct_t const&, std::tuple<int const&>, std::tuple<> >(std::piecewise_construct_t const&, std::tuple<int const&>&&, std::tuple<>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:611:23 #6 0x563e9db6efc0 in std::_Rb_tree_iterator<std::pair<int const, double> > std::_Rb_tree<int, std::pair<int const, double>, std::_Select1st<std::pair<int const, double> >, std::less<int>, std::allocator<std::pair<int const, double> > >::_M_emplace_hint_unique<std::piecewise_construct_t const&, std::tuple<int const&>, std::tuple<> >(std::_Rb_tree_const_iterator<std::pair<int const, double> >, std::piecewise_construct_t const&, std::tuple<int const&>&&, std::tuple<>&&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_tree.h:2431:19 #7 0x563e9db6ecb2 in std::map<int, double, std::less<int>, std::allocator<std::pair<int const, double> > >::operator[](int const&) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_map.h:501:15 #8 0x563e9db6ca32 in std::enable_if<(!(denc_traits<int, void>::supported)) || (!(denc_traits<double, void>::supported)), void>::type ceph::decode<int, double, std::less<int>, std::allocator<std::pair<int const, double> >, denc_traits<int, void>, denc_traits<double, void> >(std::map<int, double, std::less<int>, std::allocator<std::pair<int const, double> > >&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/include/encoding.h:1095:12 ceph#9 0x563e9db6c1d4 in ConnectionReport::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ConnectionTracker.h:37:5 ceph#10 0x563e9db6ba3c in decode(ConnectionReport&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ConnectionTracker.h:52:1 ceph#11 0x563e9db5a47e in std::enable_if<(!(denc_traits<int, void>::supported)) || (!(denc_traits<ConnectionReport, void>::supported)), void>::type ceph::decode<int, ConnectionReport, std::less<int>, std::allocator<std::pair<int const, ConnectionReport> >, denc_traits<int, void>, denc_traits<ConnectionReport, void> >(std::map<int, ConnectionReport, std::less<int>, std::allocator<std::pair<int const, ConnectionReport> > >&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/include/encoding.h:1095:5 ceph#12 0x563e9db51b69 in ConnectionTracker::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ConnectionTracker.cc:309:3 ceph#13 0x563e9da18bac in ConnectionTracker::ConnectionTracker(ceph::buffer::v15_2_0::list const&, ceph::common::CephContext*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ConnectionTracker.h:180:5 ceph#14 0x563e9d9ef57f in Election::propose_to(int, int, unsigned int, ceph::buffer::v15_2_0::list&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_election.cc:369:15 ceph#15 0x563e9da22ccb in Owner::propose_to_peers(unsigned int, ceph::buffer::v15_2_0::list&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_election.cc:145:15 ceph#16 0x563e9db2da6c in ElectionLogic::start() /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ElectionLogic.cc:143:12 ceph#17 0x563e9db2f128 in ElectionLogic::end_election_period() /home/jenkins-build/build/workspace/ceph-pull-requests/src/mon/ElectionLogic.cc:180:7 ceph#18 0x563e9da29a5d in Owner::election_timeout() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_election.cc:242:11 ceph#19 0x563e9da19936 in Owner::notify_timestep() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_election.cc:282:2 ceph#20 0x563e9d9f1181 in Election::run_timesteps(int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_election.cc:417:17 ``` in this change, we add an parameter to the handler function, so it can free the allocated `ConnectionTracker` instance. this should address the leakage reported by ASan. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

we have leak reports from ASan like ``` Direct leak of 512 byte(s) in 1 object(s) allocated from: #0 0x557c5ca1aa88 in __interceptor_calloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-dencoder+0xc0a88) (BuildId: 0431ba72332bcadbbda97c424e0d5e66a10e88d7) #1 0x7f3093a4e7e5 (<unknown module>) Direct leak of 464 byte(s) in 1 object(s) allocated from: #0 0x557c5ca1aa88 in __interceptor_calloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-dencoder+0xc0a88) (BuildId: 0431ba72332bcadbbda97c424e0d5e66a10e88d7) #1 0x7f3093a5de3e (<unknown module>) Direct leak of 248 byte(s) in 1 object(s) allocated from: #0 0x557c5ca1aa88 in __interceptor_calloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-dencoder+0xc0a88) (BuildId: 0431ba72332bcadbbda97c424e0d5e66a10e88d7) #1 0x7f30939e05a9 (<unknown module>) Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x557c5ca5566d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-dencoder+0xfb66d) (BuildId: 0431ba72332bcadbbda97c424e0d5e66a10e88d7) #1 0x7f3094c27087 (<unknown module>) #2 0x7f3094c21d64 (<unknown module>) #3 0x557c5ca680bb in DencoderPlugin::register_dencoders[abi:cxx11]() /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/ceph-dencoder/denc_plugin.h:44:5 #4 0x557c5ca5f9ea in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/ceph-dencoder/ceph_dencoder.cc:100:38 #5 0x7f309f33ad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 Direct leak of 48 byte(s) in 1 object(s) allocated from: #0 0x557c5ca5566d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-dencoder+0xfb66d) (BuildId: 0431ba72332bcadbbda97c424e0d5e66a10e88d7) #1 0x7f308c91f257 (<unknown module>) #2 0x7f308c8fd0df (<unknown module>) #3 0x557c5ca680bb in DencoderPlugin::register_dencoders[abi:cxx11]() /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/ceph-dencoder/denc_plugin.h:44:5 #4 0x557c5ca5f9ea in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/ceph-dencoder/ceph_dencoder.cc:100:38 #5 0x7f309f33ad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ``` it turns out the dencoders allocated by `register_dencoders()` are never unregistered, so let's do the proper cleanup when done with the dencoders. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

When sanitizer is enabled, unittest_rbd_mirror shows ``` ================================================================= ==1377627==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2 byte(s) in 2 object(s) allocated from: #0 0xaaaac462d088 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rbd_mirror+0x30cd088) (BuildId: 054d3b6699c622daad91c7f70e36616220fbd5ad) #1 0xaaaac522f13c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::get_client() /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:148:24 #2 0xaaaac522e60c in rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int) /root/ceph/src/tools/rbd_mirror/image_replayer/PrepareRemoteImageRequest.cc:120:5 #3 0xaaaac5232d1c in librbd::util::detail::C_CallbackAdapter<rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>, &(rbd::mirror::image_replayer::PrepareRemoteImageRequest<librbd::(anonymous namespace)::MockTestImageCtx>::handle_get_mirror_info(int))>::finish(int) /root/ceph/src/librbd/Utils.h:63:5 #4 0xaaaac4863428 in Context::complete(int) /root/ceph/src/include/Context.h:99:5 #5 0xaaaac489ac30 in librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()::operator()() const /root/ceph/src/librbd/asio/ContextWQ.h:31:12 #6 0xaaaac489aaf4 in boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>::operator()() /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:60:5 #7 0xaaaac489aaac in void boost::asio::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3 #8 0xaaaac489aa4c in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3 ceph#9 0xaaaac489a9ec in void boost::asio::detail::asio_handler_invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>*) /root/ceph/build/boost/include/boost/asio/detail/bind_handler.hpp:111:3 ceph#10 0xaaaac4899c00 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()> >(boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&, boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3 ceph#11 0xaaaac489c814 in boost::asio::detail::executor_op<boost::asio::detail::binder0<librbd::asio::ContextWQ::queue(Context*, int)::'lambda'()>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7 ceph#12 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5 ceph#13 0xaaaac489f074 in boost::asio::detail::strand_executor_service::run_ready_handlers(std::shared_ptr<boost::asio::detail::strand_executor_service::strand_impl>&) /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.ipp:150:8 ceph#14 0xaaaac489ed2c in boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>::operator()() /root/ceph/build/boost/include/boost/asio/detail/impl/strand_executor_service.hpp:136:5 ceph#15 0xaaaac489ebd4 in void boost::asio::asio_handler_invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, ...) /root/ceph/build/boost/include/boost/asio/handler_invoke_hook.hpp:88:3 ceph#16 0xaaaac489e284 in void boost_asio_handler_invoke_helpers::invoke<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void> >(boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&, boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>&) /root/ceph/build/boost/include/boost/asio/detail/handler_invoke_helpers.hpp:54:3 ceph#17 0xaaaac48a3e08 in boost::asio::detail::executor_op<boost::asio::detail::strand_executor_service::invoker<boost::asio::io_context::basic_executor_type<std::allocator<void>, 0ul> const, void>, std::allocator<void>, boost::asio::detail::scheduler_operation>::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/executor_op.hpp:71:7 ceph#18 0xaaaac489fb4c in boost::asio::detail::scheduler_operation::complete(void*, boost::system::error_code const&, unsigned long) /root/ceph/build/boost/include/boost/asio/detail/scheduler_operation.hpp:40:5 ceph#19 0xaaaac6766b1c in boost::asio::detail::scheduler::do_run_one(boost::asio::detail::conditionally_enabled_mutex::scoped_lock&, boost::asio::detail::scheduler_thread_info&, boost::system::error_code const&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:493:12 ceph#20 0xaaaac67660a4 in boost::asio::detail::scheduler::run(boost::system::error_code&) /root/ceph/build/boost/include/boost/asio/detail/impl/scheduler.ipp:210:10 ceph#21 0xaaaac7562d48 in boost::asio::io_context::run() /root/ceph/build/boost/include/boost/asio/impl/io_context.ipp:64:24 ceph#22 0xaaaac7562bcc in ceph::async::io_context_pool::start(short)::'lambda'()::operator()() const /root/ceph/src/common/async/context_pool.h:69:16 ceph#23 0xaaaac7562b18 in void std::__invoke_impl<void, ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 ceph#24 0xaaaac7562960 in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#25 0xaaaac7562804 in std::invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::invoke<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/functional:97:14 ceph#26 0xaaaac75627dc in auto std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)::operator()<ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) const /root/ceph/src/common/Thread.h:79:10 ceph#27 0xaaaac7562728 in ceph::async::io_context_pool::start(short)::'lambda'() std::__invoke_impl<void, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(std::__invoke_other, std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&)&&, ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:61:14 ceph#28 0xaaaac75624ec in std::__invoke_result<ceph::async::io_context_pool::start(short)::'lambda'()>::type std::__invoke<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()>(ceph::async::io_context_pool::start(short)::'lambda'()&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#29 0xaaaac756231c in void std::thread::_Invoker<std::tuple<std::thread make_named_thread<ceph::async::io_context_pool::start(short)::'lambda'()>(std::basic_string_view<char, std::char_traits<char> >, ceph::async::io_context_pool::start(short)::'lambda'()&&)::'lambda'(ceph::async::io_context_pool::start(short)::'lambda'()&&), ceph::async::io_context_pool::start(short)::'lambda'()> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 SUMMARY: AddressSanitizer: 2 byte(s) leaked in 2 allocation(s). ``` Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

When sanitizer is enabled, unittest_rgw_kms shows, ``` ================================================================= ==1415137==ERROR: LeakSanitizer: detected memory leaks Direct leak of 768 byte(s) in 1 object(s) allocated from: #0 0xaaaacb1381e0 in malloc (/root/ceph/build/bin/unittest_rgw_kms+0x4b81e0) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) #1 0xaaaacb1ed768 in ZeroPoolAllocator::Realloc(void*, unsigned long, unsigned long) /root/ceph/src/rgw/rgw_kms.cc:84:14 #2 0xaaaacb1ed728 in rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>* rapidjson::Realloc<rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>, ZeroPoolAllocator>(ZeroPoolAllocator&, rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>*, unsigned long, unsigned long) /root/ceph/src/s3select/rapidjson/include/rapidjson/error/../internal/../allocators.h:437:30 #3 0xaaaacb1ed4ec in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::DoReserveMembers(unsigned int, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:2260:34 #4 0xaaaacb1ed2cc in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::DoAddMember(rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:2292:13 #5 0xaaaacb1ec344 in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::AddMember(rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:1375:9 #6 0xaaaacb19769c in void add_name_val_to_obj<rapidjson::UTF8<char>, ZeroPoolAllocator>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/rgw/rgw_kms.cc:149:5 #7 0xaaaacb197058 in void add_name_val_to_obj<rapidjson::UTF8<char>, ZeroPoolAllocator>(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/rgw/rgw_kms.cc:169:3 #8 0xaaaacb1ac4a8 in TransitSecretEngine::make_actual_key(DoutPrefixProvider const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /root/ceph/src/rgw/rgw_kms.cc:485:5 ceph#9 0xaaaacb188f88 in TestSSEKMS_test_transit_makekey_Test::TestBody() /root/ceph/src/test/rgw/test_rgw_kms.cc:192:29 ceph#10 0xaaaacb34549c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#11 0xaaaacb2f864c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#12 0xaaaacb2aa958 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5 ceph#13 0xaaaacb2ac89c in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11 ceph#14 0xaaaacb2ade9c in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28 ceph#15 0xaaaacb2c9c60 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44 ceph#16 0xaaaacb34f310 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#17 0xaaaacb2ff588 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#18 0xaaaacb2c90d8 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10 ceph#19 0xaaaacb250ae4 in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#20 0xaaaacb250a60 in main /root/ceph/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#21 0xffff7cf773f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#22 0xffff7cf774c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#23 0xaaaacb0c102c in _start (/root/ceph/build/bin/unittest_rgw_kms+0x44102c) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) Direct leak of 768 byte(s) in 1 object(s) allocated from: #0 0xaaaacb1381e0 in malloc (/root/ceph/build/bin/unittest_rgw_kms+0x4b81e0) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) #1 0xaaaacb1ed768 in ZeroPoolAllocator::Realloc(void*, unsigned long, unsigned long) /root/ceph/src/rgw/rgw_kms.cc:84:14 #2 0xaaaacb1ed728 in rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>* rapidjson::Realloc<rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>, ZeroPoolAllocator>(ZeroPoolAllocator&, rapidjson::GenericMember<rapidjson::UTF8<char>, ZeroPoolAllocator>*, unsigned long, unsigned long) /root/ceph/src/s3select/rapidjson/include/rapidjson/error/../internal/../allocators.h:437:30 #3 0xaaaacb1ed4ec in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::DoReserveMembers(unsigned int, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:2260:34 #4 0xaaaacb1ed2cc in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::DoAddMember(rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:2292:13 #5 0xaaaacb1ec344 in rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>::AddMember(rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/s3select/rapidjson/include/rapidjson/document.h:1375:9 #6 0xaaaacb19769c in void add_name_val_to_obj<rapidjson::UTF8<char>, ZeroPoolAllocator>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/rgw/rgw_kms.cc:149:5 #7 0xaaaacb197058 in void add_name_val_to_obj<rapidjson::UTF8<char>, ZeroPoolAllocator>(char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, rapidjson::GenericValue<rapidjson::UTF8<char>, ZeroPoolAllocator>&, ZeroPoolAllocator&) /root/ceph/src/rgw/rgw_kms.cc:169:3 #8 0xaaaacb1b2298 in TransitSecretEngine::reconstitute_actual_key(DoutPrefixProvider const*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const&, optional_yield, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /root/ceph/src/rgw/rgw_kms.cc:569:5 ceph#9 0xaaaacb18abcc in TestSSEKMS_test_transit_reconstitutekey_Test::TestBody() /root/ceph/src/test/rgw/test_rgw_kms.cc:216:29 ceph#10 0xaaaacb34549c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#11 0xaaaacb2f864c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#12 0xaaaacb2aa958 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5 ceph#13 0xaaaacb2ac89c in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11 ceph#14 0xaaaacb2ade9c in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28 ceph#15 0xaaaacb2c9c60 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44 ceph#16 0xaaaacb34f310 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#17 0xaaaacb2ff588 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#18 0xaaaacb2c90d8 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10 ceph#19 0xaaaacb250ae4 in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#20 0xaaaacb250a60 in main /root/ceph/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#21 0xffff7cf773f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#22 0xffff7cf774c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#23 0xaaaacb0c102c in _start (/root/ceph/build/bin/unittest_rgw_kms+0x44102c) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) ... ... Indirect leak of 18 byte(s) in 1 object(s) allocated from: #0 0xaaaab6e3f148 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_kms+0x4ef148) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) #1 0xffffac4a0fa0 in __gnu_cxx::new_allocator<char>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0xffffac4a0f1c in std::allocator<char>::allocate(unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0xffffac4a0f1c in std::allocator_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20 #4 0xffffac4a0c1c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:153:14 #5 0xffffac4a0618 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*, std::forward_iterator_tag) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.tcc:219:14 #6 0xffffac4a0398 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct_aux<char*>(char*, char*, std::__false_type) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:255:11 #7 0xffffac4a00d0 in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<char*>(char*, char*) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:274:4 #8 0xffffac49fe0c in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/basic_string.h:459:9 ceph#9 0xffffacc9b490 in std::enable_if<((__exactly_once<std::variant_alternative<__accepted_index<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d> >::type>) && (is_constructible_v<std::variant_alternative<__accepted_index<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d> >::type, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>)) && (is_assignable_v<std::variant_alternative<__accepted_index<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d> >::type&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>), std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d>&>::type std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d>::operator=<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/variant:1477:26 ceph#10 0xffffacc9298c in Option::parse_value(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::variant<std::monostate, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long, long, double, bool, entity_addr_t, entity_addrvec_t, std::chrono::duration<long, std::ratio<1l, 1l> >, std::chrono::duration<long, std::ratio<1l, 1000l> >, Option::size_t, uuid_d>*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) const /root/ceph/src/common/options.cc:165:10 ceph#11 0xffffacb813cc in md_config_t::_set_val(ConfigValues&, ConfigTracker const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, Option const&, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >*) /root/ceph/src/common/config.cc:1395:15 ceph#12 0xffffacb8f084 in md_config_t::set_val(ConfigValues&, ConfigTracker const&, std::basic_string_view<char, std::char_traits<char> >, char const*, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) /root/ceph/src/common/config.cc:930:13 ceph#13 0xaaaab6ecb014 in md_config_t::set_val(ConfigValues&, ConfigTracker const&, std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) /root/ceph/src/common/config.h:180:12 ceph#14 0xaaaab6e72464 in ceph::common::ConfigProxy::set_val(std::basic_string_view<char, std::char_traits<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >*) /root/ceph/src/common/config_proxy.h:239:19 ceph#15 0xaaaab6e53bf8 in TestSSEKMS_non_existent_vault_token_file_Test::TestBody() /root/ceph/src/test/rgw/test_rgw_kms.cc:84:14 ceph#16 0xaaaab701549c in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#17 0xaaaab6fc864c in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#18 0xaaaab6f7a958 in testing::Test::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2680:5 ceph#19 0xaaaab6f7c89c in testing::TestInfo::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:2858:11 ceph#20 0xaaaab6f7de9c in testing::TestSuite::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:3012:28 ceph#21 0xaaaab6f99c60 in testing::internal::UnitTestImpl::RunAllTests() /root/ceph/src/googletest/googletest/src/gtest.cc:5723:44 ceph#22 0xaaaab701f310 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2605:10 ceph#23 0xaaaab6fcf588 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /root/ceph/src/googletest/googletest/src/gtest.cc:2641:14 ceph#24 0xaaaab6f990d8 in testing::UnitTest::Run() /root/ceph/src/googletest/googletest/src/gtest.cc:5306:10 ceph#25 0xaaaab6f20ae4 in RUN_ALL_TESTS() /root/ceph/src/googletest/googletest/include/gtest/gtest.h:2486:46 ceph#26 0xaaaab6f20a60 in main /root/ceph/src/googletest/googlemock/src/gmock_main.cc:70:10 ceph#27 0xffffa99573f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#28 0xffffa99574c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#29 0xaaaab6d9102c in _start (/root/ceph/build/bin/unittest_rgw_kms+0x44102c) (BuildId: 2005acbecb5cbd5bc928c98209bd613f20ecbacf) SUMMARY: AddressSanitizer: 112735554 byte(s) leaked in 36910 allocation(s). ``` 1. fix memory reallocate issue 2. free cct Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

When sanitizer is enabled, unittest_erasure_code_shec shows: ``` ================================================================= ==437976==ERROR: LeakSanitizer: detected memory leaks Direct leak of 2021708 byte(s) in 29731 object(s) allocated from: #0 0xaaaab0144820 in malloc (/root/ceph-19.0.0/build/bin/unittest_erasure_code_shec+0x1d4820) (BuildId: c0999ecf82504ed7e184ea4b4b9ae9d32faa1c46) #1 0xffffa770057c in crush_make_rule /root/ceph-19.0.0/src/crush/builder.c:104:9 #2 0xffffa7751638 in CrushWrapper::add_simple_rule_at(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, int, std::ostream*) /root/ceph-19.0.0/src/crush/CrushWrapper.cc:2327:22 #3 0xffffa7751d2c in CrushWrapper::add_simple_rule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, int, std::ostream*) /root/ceph-19.0.0/src/crush/CrushWrapper.cc:2369:10 #4 0xffffa39d6198 in ceph::ErasureCode::create_rule(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, CrushWrapper&, std::ostream*) const /root/ceph-19.0.0/src/erasure-code/ErasureCode.cc:76:18 #5 0xaaaab01f5a6c in thread3(void*) /root/ceph-19.0.0/src/test/erasure-code/TestErasureCodeShec.cc:2756:11 #6 0xffffa2dad5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 #7 0xffffa2e15ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 SUMMARY: AddressSanitizer: 2021708 byte(s) leaked in 29731 allocation(s). ``` When crush_add_rule exceeds the number of max_rules, crush_rule shoule be freed right before returning in caller site. Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

When sanitizer is enabled, unittest_rgw_amqp shows, ``` ================================================================= ==1429129==ERROR: LeakSanitizer: detected memory leaks Direct leak of 416 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57eecfc in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:12 #2 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f4d88 in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f4204 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_ack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3728 in boost::lockfree::queue<amqp_basic_ack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2ea8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:90:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f90bc in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f8538 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_nack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3a6c in boost::lockfree::queue<amqp_basic_nack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2eb8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:91:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 9 byte(s) in 1 object(s) allocated from: #0 0xaaaab56690a0 in malloc (/root/ceph/build/bin/unittest_rgw_amqp+0x1890a0) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f2754 in amqp_bytes_malloc_dup /root/ceph/src/test/rgw/amqp_mock.cc:384:18 #2 0xaaaab580b4b4 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:509:28 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 65536 byte(s) in 1024 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f4d88 in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f4204 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_ack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3728 in boost::lockfree::queue<amqp_basic_ack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2ea8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:90:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 65536 byte(s) in 1024 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f90bc in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f8538 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_nack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3a6c in boost::lockfree::queue<amqp_basic_nack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2eb8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:91:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57eefb0 in amqp_tcp_socket_new /root/ceph/src/test/rgw/amqp_mock.cc:127:19 #2 0xaaaab5809740 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:401:14 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f102c in amqp_queue_declare /root/ceph/src/test/rgw/amqp_mock.cc:283:18 #2 0xaaaab580ad14 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:480:27 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f2280 in amqp_basic_consume /root/ceph/src/test/rgw/amqp_mock.cc:359:20 #2 0xaaaab580b124 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:493:29 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0214 in amqp_channel_open /root/ceph/src/test/rgw/amqp_mock.cc:213:23 #2 0xaaaab5809e78 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:448:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0294 in amqp_channel_open /root/ceph/src/test/rgw/amqp_mock.cc:217:21 #2 0xaaaab580a188 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:453:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 1 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f1454 in amqp_confirm_select /root/ceph/src/test/rgw/amqp_mock.cc:291:20 #2 0xaaaab580a49c in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:458:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 1 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0548 in amqp_exchange_declare /root/ceph/src/test/rgw/amqp_mock.cc:231:21 #2 0xaaaab580a93c in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:466:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 SUMMARY: AddressSanitizer: 131723 byte(s) leaked in 2059 allocation(s). ``` So to prevent multiple threads from operating the same element at the same time, add the lock only to erase and ensure fully create connection before emplace. Fixes: https://tracker.ceph.com/issues/66266 Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

When sanitizer is enabled, a part of 'run-cli-tests' output shows, ``` ================================================================= ==1263095==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60c00000c000 at pc 0x7f80a4b0a040 bp 0x7ffe3176d550 sp 0x7ffe3176d548 READ of size 8 at 0x60c00000c000 thread T0 #0 0x7f80a4b0a03f in CrushWrapper::get_new_bucket_id() /home/jenkins-build/build/workspace/ceph-pull-requests/src/crush/CrushWrapper.cc:2189:10 #1 0x7f80a4b03f20 in CrushWrapper::reclassify(ceph::common::CephContext*, std::ostream&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/crush/CrushWrapper.cc:1957:20 #2 0x55d567dfbcec in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/crushtool.cc:1215:19 #3 0x7f80a06c7d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 #4 0x7f80a06c7e3f in __libc_start_main csu/../csu/libc-start.c:392:3 #5 0x55d567d2b4d4 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/crushtool+0xb54d4) (BuildId: ce3df2d268a883ca3965158085f32e534cbedaf5) 0x60c00000c000 is located 0 bytes to the right of 128-byte region [0x60c00000bf80,0x60c00000c000) allocated by thread T0 here: #0 0x55d567dae508 in __interceptor_calloc (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/crushtool+0x138508) (BuildId: ce3df2d268a883ca3965158085f32e534cbedaf5) #1 0x7f80a4b164cf in CrushWrapper::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/crush/CrushWrapper.cc:3267:38 #2 0x55d567df69eb in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/tools/crushtool.cc:919:13 #3 0x7f80a06c7d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 SUMMARY: AddressSanitizer: heap-buffer-overflow /home/jenkins-build/build/workspace/ceph-pull-requests/src/crush/CrushWrapper.cc:2189:10 in CrushWrapper::get_new_bucket_id() ``` fixes: https://tracker.ceph.com/issues/66861 Signed-off-by: Rongqi Sun <sunrongqi@huawei.com>

When sanitizer is enabled, unittest_rgw_amqp shows, ``` ================================================================= ==1429129==ERROR: LeakSanitizer: detected memory leaks Direct leak of 416 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57eecfc in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:12 #2 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f4d88 in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f4204 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_ack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3728 in boost::lockfree::queue<amqp_basic_ack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2ea8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:90:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 64 byte(s) in 1 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f90bc in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f8538 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_nack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3a6c in boost::lockfree::queue<amqp_basic_nack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2eb8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:91:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Direct leak of 9 byte(s) in 1 object(s) allocated from: #0 0xaaaab56690a0 in malloc (/root/ceph/build/bin/unittest_rgw_amqp+0x1890a0) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f2754 in amqp_bytes_malloc_dup /root/ceph/src/test/rgw/amqp_mock.cc:384:18 #2 0xaaaab580b4b4 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:509:28 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 65536 byte(s) in 1024 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f4d88 in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f4204 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_ack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_ack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3728 in boost::lockfree::queue<amqp_basic_ack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2ea8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:90:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 65536 byte(s) in 1024 object(s) allocated from: #0 0xaaaab5669bb8 in posix_memalign (/root/ceph/build/bin/unittest_rgw_amqp+0x189bb8) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f5294 in boost::alignment::aligned_alloc(unsigned long, unsigned long) /root/ceph/build/boost/include/boost/align/detail/aligned_alloc_posix.hpp:26:9 #2 0xaaaab57f90bc in boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul>::allocate(unsigned long, void const*) /root/ceph/build/boost/include/boost/align/aligned_allocator.hpp:70:19 #3 0xaaaab57f8538 in boost::lockfree::detail::freelist_stack<boost::lockfree::queue<amqp_basic_nack_t_>::node, boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >::freelist_stack<boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> >(boost::alignment::aligned_allocator<boost::lockfree::queue<amqp_basic_nack_t_>::node, 64ul> const&, unsigned long) /root/ceph/build/boost/include/boost/lockfree/detail/freelist.hpp:62:31 #4 0xaaaab57f3a6c in boost::lockfree::queue<amqp_basic_nack_t_>::queue(unsigned long) /root/ceph/build/boost/include/boost/lockfree/queue.hpp:234:9 #5 0xaaaab57f2eb8 in amqp_connection_state_t_::amqp_connection_state_t_() /root/ceph/src/test/rgw/amqp_mock.cc:91:5 #6 0xaaaab57eed04 in amqp_new_connection /root/ceph/src/test/rgw/amqp_mock.cc:110:16 #7 0xaaaab58095d8 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:373:16 #8 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 ceph#9 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 ceph#10 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 ceph#11 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 ceph#12 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 ceph#13 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#14 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#15 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#16 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57eefb0 in amqp_tcp_socket_new /root/ceph/src/test/rgw/amqp_mock.cc:127:19 #2 0xaaaab5809740 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:401:14 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 24 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f102c in amqp_queue_declare /root/ceph/src/test/rgw/amqp_mock.cc:283:18 #2 0xaaaab580ad14 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:480:27 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f2280 in amqp_basic_consume /root/ceph/src/test/rgw/amqp_mock.cc:359:20 #2 0xaaaab580b124 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:493:29 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0214 in amqp_channel_open /root/ceph/src/test/rgw/amqp_mock.cc:213:23 #2 0xaaaab5809e78 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:448:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 16 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0294 in amqp_channel_open /root/ceph/src/test/rgw/amqp_mock.cc:217:21 #2 0xaaaab580a188 in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:453:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 1 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f1454 in amqp_confirm_select /root/ceph/src/test/rgw/amqp_mock.cc:291:20 #2 0xaaaab580a49c in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:458:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 Indirect leak of 1 byte(s) in 1 object(s) allocated from: #0 0xaaaab56a0008 in operator new(unsigned long) (/root/ceph/build/bin/unittest_rgw_amqp+0x1c0008) (BuildId: a20c317434e8d5f2ec33bbb71a69d81eb751c494) #1 0xaaaab57f0548 in amqp_exchange_declare /root/ceph/src/test/rgw/amqp_mock.cc:231:21 #2 0xaaaab580a93c in rgw::amqp::new_state(rgw::amqp::connection_t*, rgw::amqp::connection_id_t const&) /root/ceph/src/rgw/rgw_amqp.cc:466:21 #3 0xaaaab5813c70 in rgw::amqp::Manager::run() /root/ceph/src/rgw/rgw_amqp.cc:684:18 #4 0xaaaab5849e50 in void std::__invoke_impl<void, void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(std::__invoke_memfun_deref, void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:74:14 #5 0xaaaab5849b48 in std::__invoke_result<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>::type std::__invoke<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*>(void (rgw::amqp::Manager::*&&)() noexcept, rgw::amqp::Manager*&&) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/invoke.h:96:14 #6 0xaaaab5849978 in void std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::_M_invoke<0ul, 1ul>(std::_Index_tuple<0ul, 1ul>) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:259:13 #7 0xaaaab584979c in std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> >::operator()() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:266:11 #8 0xaaaab5849420 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<void (rgw::amqp::Manager::*)() noexcept, rgw::amqp::Manager*> > >::_M_run() /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/std_thread.h:211:13 ceph#9 0xffffb0fb31f8 (/lib/aarch64-linux-gnu/libstdc++.so.6+0xd31f8) (BuildId: a012b2bb77110e84b266cd7425b50e57427abb02) ceph#10 0xffffb0d7d5c4 in start_thread nptl/./nptl/pthread_create.c:442:8 ceph#11 0xffffb0de5ed8 misc/../sysdeps/unix/sysv/linux/aarch64/clone.S:79 SUMMARY: AddressSanitizer: 131723 byte(s) leaked in 2059 allocation(s). ``` So to prevent multiple threads from operating the same element at the same time, add the lock only to erase and ensure fully create connection before emplace. Fixes: https://tracker.ceph.com/issues/66266 Signed-off-by: Rongqi Sun <sunrongqi@huawei.com> (cherry picked from commit 958ecba)

…n async task test_messenger_thrash UT shows, ``` ==461141==ERROR: AddressSanitizer: stack-use-after-return on address 0xffffb0b37c20 at pc 0xaaaad7239508 bp 0xffffeb113c50 sp 0xffffeb113c48 READ of size 4 at 0xffffb0b37c20 thread T0 #0 0xaaaad7239504 in (anonymous namespace)::SyntheticWorkload::wait_for_done()::'lambda0'()::operator()() const /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/crimson/test_messenger_thrash.cc:455:13 #1 0xaaaad723a1c0 in seastar::internal::do_until_state<(anonymous namespace)::SyntheticWorkload::wait_for_done()::'lambda'(), (anonymous namespace)::SyntheticWorkload::wait_for_done()::'lambda0'()>::run_and_dispose() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/include/seastar/core/loop.hh:303:26 #2 0xaaaadacfb790 in seastar::reactor::run_tasks(seastar::reactor::task_queue&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/reactor.cc:2653:14 #3 0xaaaadad04288 in seastar::reactor::run_some_tasks() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/reactor.cc:3123:9 #4 0xaaaadad07cd0 in seastar::reactor::do_run() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/reactor.cc:3291:9 #5 0xaaaadad05d60 in seastar::reactor::run() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/reactor.cc:3181:16 #6 0xaaaadaa860d8 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/app-template.cc:276:31 #7 0xaaaadaa83fb0 in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/seastar/src/core/app-template.cc:167:12 #8 0xaaaad7203d88 in main /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/crimson/test_messenger_thrash.cc:669:14 ceph#9 0xffffb32773f8 in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16 ceph#10 0xffffb32774c8 in __libc_start_main csu/../csu/libc-start.c:392:3 ceph#11 0xaaaad712546c in _start (/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/bin/unittest-seastar-messenger-thrash+0x3b5546c) (BuildId: b0048d750e057d178816f94b3ce0459971785191) ``` Address 0xffffb0b37c20 is located in stack of thread T0 at offset 32 in frame #0 0xaaaad7493ed8 in ceph::buffer::v15_2_0::list::buffers_t::clear_and_dispose() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/include/buffer.h:596 Signed-off-by: arm7star <arm7star@qq.com>

https://vote.heliosvoting.org/helios/elections/e03494ce-e04c-41d0-bb05-ec5ccc632ce4/view Question #1 Update election requirements for Ceph Executive Council Elections? Remove "ranked-choice" requirement 13 Keep "ranked-choice" requirement (no change) 16 Question #2 Require periodic elections in governance charter? No (no change) 8 Annual 15 Semi-annual 3 Quarterly 2 Question #3 Update the Ceph Executive Council term length? Change to 3 years 14 Keep 2 years (no change) 14 Question #4 Amend governance document to require a supermajority of votes for amendments to the governance model? The current requirement is a simple majority. Require a supermajority 20 Require a simple majority (no change) 9 Question #5 Clarify "supermajority" and "majority" election requirements? Of members voting on a given question (abstaining does not bias the vote) 18 Of members voting on the election (abstaining is an implicit "no") 6 Of members in the CSC 3 Signed-off-by: Patrick Donnelly <pdonnell@ibm.com>

before this change, we allocate coefficients table with malloc() in ErasureCodeIsaDefault::prepare(), but free them using delete. so, in this change, we use new [] and delete [] to allocate and free them, to be more consistent, and to silence this warning. this issue was identified by AddressSanitizer: ``` 2025-03-22T04:27:58.210 INFO:tasks.ceph.osd.3.smithi102.stderr:==42899==ERROR: AddressSanitizer: alloc-dealloc-mismatch (operator new [] vs free) on 0x611000570d40 2025-03-22T04:27:58.253 DEBUG:teuthology.orchestra.run.smithi102:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 env ASAN_OPTIONS=detect_leaks=0,detect_odr_violation=0,alloc_dealloc_mismatch=0 LD_PRELOAD=/lib64/libasan.so.6 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #0 0x7efdc30b46b7 in free (/lib64/libasan.so.6+0xb46b7) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #1 0x7efd9e2e5f26 in ErasureCodeIsaTableCache::setEncodingTable(int, int, int, unsigned char*) (/usr/lib64/ceph/erasure-code/libec_isa.so+0x3bf26) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #2 0x7efd9e2f0fae in ErasureCodeIsaDefault::prepare() (/usr/lib64/ceph/erasure-code/libec_isa.so+0x46fae) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #3 0x7efd9e2bfa3e in ErasureCodeIsa::init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream*) (/usr/lib64/ceph/erasure-code/libec_isa.so+0x15a3e) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #4 0x7efd9e2ff510 in ErasureCodePluginIsa::factory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::shared_ptr<ceph::ErasureCodeInterface>*, std::ostream*) (/usr/lib64/ceph/erasure-code/libec_isa.so+0x55510) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #5 0x55a7ef19b395 in ceph::ErasureCodePluginRegistry::factory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::shared_ptr<ceph::ErasureCodeInterface>*, std::ostream*) (/usr/bin/ceph-osd+0x4dcb395) 2025-03-22T04:27:58.442 INFO:tasks.ceph.osd.3.smithi102.stderr: #6 0x55a7eb9089d8 in PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, ceph::common::CephContext*) (/usr/bin/ceph-osd+0x15389d8) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: #7 0x55a7eb4b63ec in PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, spg_t) (/usr/bin/ceph-osd+0x10e63ec) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: #8 0x55a7eaef1cc7 in OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t) (/usr/bin/ceph-osd+0xb21cc7) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#9 0x55a7eaef8d69 in OSD::handle_pg_create_info(std::shared_ptr<OSDMap const> const&, PGCreateInfo const*) (/usr/bin/ceph-osd+0xb28d69) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#10 0x55a7eafeb9ed in OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (/usr/bin/ceph-osd+0xc1b9ed) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#11 0x55a7ed04c9aa in ShardedThreadPool::shardedthreadpool_worker(unsigned int) (/usr/bin/ceph-osd+0x2c7c9aa) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#12 0x55a7ed04d338 in ShardedThreadPool::WorkThreadSharded::entry() (/usr/bin/ceph-osd+0x2c7d338) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#13 0x55a7ecf92b46 in Thread::entry_wrapper() (/usr/bin/ceph-osd+0x2bc2b46) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#14 0x55a7ecf92b82 in Thread::_entry_func(void*) (/usr/bin/ceph-osd+0x2bc2b82) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#15 0x7efdc1e8a3b1 in start_thread (/lib64/libc.so.6+0x8a3b1) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#16 0x7efdc1f0f42f in __GI___clone3 (/lib64/libc.so.6+0x10f42f) 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr: 2025-03-22T04:27:58.443 INFO:tasks.ceph.osd.3.smithi102.stderr:0x611000570d40 is located 0 bytes inside of 256-byte region [0x611000570d40,0x611000570e40) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr:allocated by thread T817 here: 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #0 0x7efdc30b64d7 in operator new[](unsigned long) (/lib64/libasan.so.6+0xb64d7) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #1 0x7efd9e2f0e32 in ErasureCodeIsaDefault::prepare() (/usr/lib64/ceph/erasure-code/libec_isa.so+0x46e32) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #2 0x7efd9e2bfa3e in ErasureCodeIsa::init(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::ostream*) (/usr/lib64/ceph/erasure-code/libec_isa.so+0x15a3e) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #3 0x7efd9e2ff510 in ErasureCodePluginIsa::factory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::shared_ptr<ceph::ErasureCodeInterface>*, std::ostream*) (/usr/lib64/ceph/erasure-code/libec_isa.so+0x55510) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #4 0x55a7ef19b395 in ceph::ErasureCodePluginRegistry::factory(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::shared_ptr<ceph::ErasureCodeInterface>*, std::ostream*) (/usr/bin/ceph-osd+0x4dcb395) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #5 0x55a7eb9089d8 in PGBackend::build_pg_backend(pg_pool_t const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, PGBackend::Listener*, coll_t, boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore*, ceph::common::CephContext*) (/usr/bin/ceph-osd+0x15389d8) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #6 0x55a7eb4b63ec in PrimaryLogPG::PrimaryLogPG(OSDService*, std::shared_ptr<OSDMap const>, PGPool const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > const&, spg_t) (/usr/bin/ceph-osd+0x10e63ec) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #7 0x55a7eaef1cc7 in OSD::_make_pg(std::shared_ptr<OSDMap const>, spg_t) (/usr/bin/ceph-osd+0xb21cc7) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: #8 0x55a7eaef8d69 in OSD::handle_pg_create_info(std::shared_ptr<OSDMap const> const&, PGCreateInfo const*) (/usr/bin/ceph-osd+0xb28d69) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#9 0x55a7eafeb9ed in OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*) (/usr/bin/ceph-osd+0xc1b9ed) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#10 0x55a7ed04c9aa in ShardedThreadPool::shardedthreadpool_worker(unsigned int) (/usr/bin/ceph-osd+0x2c7c9aa) 2025-03-22T04:27:58.444 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#11 0x55a7ed04d338 in ShardedThreadPool::WorkThreadSharded::entry() (/usr/bin/ceph-osd+0x2c7d338) 2025-03-22T04:27:58.445 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#12 0x55a7ecf92b46 in Thread::entry_wrapper() (/usr/bin/ceph-osd+0x2bc2b46) 2025-03-22T04:27:58.445 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#13 0x55a7ecf92b82 in Thread::_entry_func(void*) (/usr/bin/ceph-osd+0x2bc2b82) 2025-03-22T04:27:58.445 INFO:tasks.ceph.osd.3.smithi102.stderr: ceph#14 0x7efdc1e8a3b1 in start_thread (/lib64/libc.so.6+0x8a3b1) 2025-03-22T04:27:58.445 INFO:tasks.ceph.osd.3.smithi102.stderr: ``` Fixes: https://tracker.ceph.com/issues/70619 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, in __ceph_abort and related abort handlers, we allocated ClibBackTrace instances using raw pointers without proper cleanup. Since these handlers terminate execution, the leaks didn't affect production systems but were correctly flagged by ASan during testing: ``` Direct leak of 288 byte(s) in 1 object(s) allocated from: #0 0x55aefe8cb65d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_ceph_assert+0x1f465d) (BuildId: a4faeddac80b0d81062bd53ede3388c0c10680bc) #1 0x7f3b84da988d in ceph::__ceph_assertf_fail(char const*, char const*, int, char const*, char const*, ...) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/assert.cc:157:21 #2 0x55aefe8cf04b in supressed_assertf_line22() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/ceph_assert.cc:22:3 #3 0x55aefe8ce4e6 in CephAssertDeathTest_ceph_assert_supresssions_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/ceph_assert.cc:31:3 #4 0x55aefe99135d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2653:10 #5 0x55aefe94f015 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2689:14 ... ``` This commit resolves the issue by using std::unique_ptr to manage the lifecycle of backtrace objects, ensuring proper cleanup even in non-returning functions. While these leaks had no practical impact in production (as the process terminates anyway), fixing them improves code quality and eliminates false positives in memory analysis tools. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

…ms in destructor Fix memory leak in test_not_before_queue identified by AddressSanitizer. Previously, the test was terminating without properly dequeueing all elements, causing resource leaks during teardown. This change implements a proper cleanup mechanism that: 1. Advances the queue's time until all elements become eligible for processing 2. Dequeues all remaining elements to ensure proper destruction 3. Guarantees clean teardown even for elements with future timestamps The fix eliminates ASan-reported leaks occurring in the not_before_queue_t::enqueue method when test cases were torn down prematurely. A sample of the error from ASan: ``` Direct leak of 1800 byte(s) in 15 object(s) allocated from: #0 0x7f71b1f1ab5b in operator new(unsigned long) ../../../../src/libsanitizer/asan/asan_new_delete.cpp:86 #1 0x55fb4c977058 in void not_before_queue_t<tv_t, test_time_t>::enqueue<tv_t const&>(tv_t const&) /home/kefu/dev/ceph/src/common/not_before_queue.h:164 #2 0x55fb4c97748b in NotBeforeTest::load_test_data(std::vector<tv_t, std::allocator<tv_t> > const&) /home/kefu/dev/ceph/src/test/test_not_before_queue.cc:67 #3 0x55fb4c961112 in NotBeforeTest_RemoveIfByClass_no_cond_Test::TestBody() /home/kefu/dev/ceph/src/test/test_not_before_queue.cc:213 #4 0x55fb4ca05f67 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 #5 0x55fb4ca1c4f7 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2689 #6 0x55fb4c9e1104 in testing::Test::Run() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2728 #7 0x55fb4c9e16e2 in testing::TestInfo::Run() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2874 #8 0x55fb4c9e73b4 in testing::TestSuite::Run() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:3052 ceph#9 0x55fb4c9f059b in testing::internal::UnitTestImpl::RunAllTests() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:6004 ceph#10 0x55fb4ca064ff in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 ceph#11 0x55fb4ca1d1bf in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2689 ceph#12 0x55fb4c9e124d in testing::UnitTest::Run() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:5583 ceph#13 0x55fb4c97a0b6 in RUN_ALL_TESTS() /home/kefu/dev/ceph/src/googletest/googletest/include/gtest/gtest.h:2334 ceph#14 0x55fb4c979ffc in main /home/kefu/dev/ceph/src/googletest/googlemock/src/gmock_main.cc:71 ceph#15 0x7f71ae833ca7 in __libc_start_call_main ../sysdeps/nptl/libc_start_call_main.h:58 SUMMARY: AddressSanitizer: 1800 byte(s) leaked in 15 allocation(s). ``` Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, we had memory leak in the test_bluestore_types.cc tests where `BufferCacheShard` and `OnodeCacheShard` objects were allocated with raw pointers but never freed, causing leaks detected by AddressSanitizer. ASan rightly pointed this out: ``` Direct leak of 224 byte(s) in 1 object(s) allocated from: #0 0x55a7432a079d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluestore_types+0xf2e79d) (BuildId: c3bec647afa97df6bb147bc82eac937531fc6272) #1 0x55a743523340 in BlueStore::BufferCacheShard::create(BlueStore*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, ceph::common::PerfCounters*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/Bl ueStore.cc:1678:9 #2 0x55a74330b617 in ExtentMap_seek_lextent_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluestore_types.cc:1077:7 #3 0x55a7434f2b2d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest. cc:2653:10 #4 0x55a7434b5775 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc: 2689:14 #5 0x55a74347005d in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2728:5 ``` ``` Direct leak of 9928 byte(s) in 1 object(s) allocated from: #0 0x7ff249d21a2d in operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_new_delete.cpp:86 #1 0x6048ed878b76 in BlueStore::OnodeCacheShard::create(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::common::PerfCounters*) /home/kefu/dev/ceph/src/os/bluestore/BlueStore.cc:1219 #2 0x6048ed66d4f9 in GarbageCollector_BasicTest_Test::TestBody() /home/kefu/dev/ceph/src/test/objectstore/test_bluestore_types.cc:2662 #3 0x6048ed820555 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 #4 0x6048ed80c78a in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2689 #5 0x6048ed7b8bfa in testing::Test::Run() /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2728 ``` In this change, we replace raw pointer allocation with unique_ptr to ensure automatic cleanup when the objects go out of scope. ` Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Memory leak was detected by AddressSanitizer in dbstore tests. Objects inserted into a static objectmap were not being properly freed when tests completed without explicitly deleting buckets. The leak occurred because: - Objects were preserved in a static map after insertion - Objects were only freed by DB::objectmapDelete() when deleting the corresponding bucket - In tests, buckets weren't always deleted after insertion, leaving objects allocated ASan rightly pointed this out: ``` Direct leak of 200 byte(s) in 1 object(s) allocated from: #0 0x55c420f5274d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_dbstore_tests+0x4df74d) (BuildId: c19ed2d2b1ead306fdc59fc311f546a287300010) #1 0x55c42143cfaf in SQLGetBucket::Execute(DoutPrefixProvider const*, rgw::store::DBOpParams*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/driver/dbstore/sqlite/sqliteDB.cc:1689:11 #2 0x55c42102751c in rgw::store::DB::ProcessOp(DoutPrefixProvider const*, std::basic_string_view<char, std::char_traits<char>>, rgw::store::DBOpParams*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/driver/dbstore/common/dbstore.cc:251:16 #3 0x55c4210328c9 in rgw::store::DB::get_bucket_info(DoutPrefixProvider const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, RGWBucketInfo&, std::map<std::__cxx11 ::basic_string<char, std::char_traits<char>, std::allocator<char>>, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::c har_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, ceph::buffer::v15_2_0::list>>>*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l>>>*, obj_version*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/driver/dbstore/common/dbstore.cc:450:9 #4 0x55c421034357 in rgw::store::DB::create_bucket(DoutPrefixProvider const*, std::variant<rgw_user, rgw_account_id> const&, rgw_bucket const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const&, rgw_placement_rule const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>> const, ceph::buffer::v15_2_0::list>>> const&, std::optional<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>> const&, std::optional<RGWQuotaInfo> const&, std::optional<std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l>>>>, obj_version*, RGWBucketInfo&, optional_yield) /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/driver/dbstore/common/dbstore.cc:504:9 #5 0x55c420f6c3ed in DBStoreTest_CreateBucket_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/rgw/driver/dbstore/tests/dbstore_tests.cc:495:13 #6 0x55c42174fa0d in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2653:10 #7 0x55c421717f25 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2689:14 #8 0x55c4216d4bbd in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2728:5 ``` In this change, we: - Free objectmap entries in DB::Destroy() to ensure cleanup on shutdown - Call DB::Destroy() in DBStoreTest::TearDown() to guarantee cleanup after each test Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, error messages passed to luaL_error() were formatted using std::string concatenation. Since luaL_error() never returns (it throws a Lua exception via longjmp), the allocated std::string memory was leaked, as detected by AddressSanitizer: ``` Direct leak of 105 byte(s) in 1 object(s) allocated from: #0 0x7fc5f1921a2d in operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_new_delete.cpp:86 #1 0x563bd89144c7 in std::__new_allocator<char>::allocate(unsigned long, void const*) /usr/include/c++/15.1.1/bits/new_allocator.h:151 #2 0x563bd89144c7 in std::allocator<char>::allocate(unsigned long) /usr/include/c++/15.1.1/bits/allocator.h:203 #3 0x563bd89144c7 in std::allocator_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) /usr/include/c++/15.1.1/bits/alloc_traits.h:614 #4 0x563bd89144c7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_allocate(std::allocator<char>&, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.h:142 #5 0x563bd89144c7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.tcc:164 #6 0x563bd896ae1b in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.tcc:363 #7 0x563bd896b256 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_append(char const*, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.tcc:455 #8 0x563bd896b2bb in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::append(char const*) /usr/include/c++/15.1.1/bits/basic_string.h:1585 ceph#9 0x563bd943c2f2 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > std::operator+<char, std::char_traits<char>, std::allocator<char> >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&&, char const*) /usr/include/c++/15.1.1/bits/basic_string.h:3977 ceph#10 0x563bd943c2f2 in rgw::lua::lua_state_guard::runtime_hook(lua_State*, lua_Debug*) /home/kefu/dev/ceph/src/rgw/rgw_lua_utils.cc:245 ceph#11 0x7fc5f139f8ef (/usr/lib/liblua.so.5.4+0xe8ef) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#12 0x7fc5f139fbfe (/usr/lib/liblua.so.5.4+0xebfe) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#13 0x7fc5f13b26fe (/usr/lib/liblua.so.5.4+0x216fe) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#14 0x7fc5f139f581 (/usr/lib/liblua.so.5.4+0xe581) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#15 0x7fc5f139b735 (/usr/lib/liblua.so.5.4+0xa735) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#16 0x7fc5f139ba8f (/usr/lib/liblua.so.5.4+0xaa8f) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#17 0x7fc5f139f696 in lua_pcallk (/usr/lib/liblua.so.5.4+0xe696) (BuildId: b7533e2973d4b0d82e10fc29973ec5a8d355d2b8) ceph#18 0x563bd8a925ef in rgw::lua::request::execute(rgw::sal::Driver*, RGWREST*, OpsLogSink*, req_state*, RGWOp*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /home/kefu/dev/ceph/src/rgw/rgw_lua_request.cc:824 ceph#19 0x563bd8952e3d in TestRGWLua_LuaRuntimeLimit_Test::TestBody() /home/kefu/dev/ceph/src/test/rgw/test_rgw_lua.cc:1628 ceph#20 0x563bd8a37817 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 ceph#21 0x563bd8a509b5 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) (/home/kefu/dev/ceph/build/bin/unittest_rgw_lua+0x11199b5) (BuildId: b2628caba5290d882d25f7bea166f058b682bc85)` ``` This change replaces std::string formatting with stack-allocated buffer and std::to_chars() to eliminate the memory leak. Note: We cannot format int64_t directly through luaL_error() because lua_pushfstring() does not support long long or int64_t format specifiers, even in Lua 5.4 (see https://www.lua.org/manual/5.4/manual.html#lua_pushfstring). Since libstdc++ uses int64_t for std::chrono::milliseconds::rep, we use std::to_chars() for safe, efficient conversion without heap allocation. The maximum runtime limit was a configuration introduced by 3e3cb15. Fixes: https://tracker.ceph.com/issues/71595 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, we had Resource leak in Directory::for_each() where directory streams opened with fdopendir() were never properly closed, causing a 32KB leak per unclosed directory handle. In this change, we add closedir() call to properly release directory stream resources after enumeration completes. ASan report: ``` Direct leak of 32816 byte(s) in 1 object(s) allocated from: #0 0x738387320e15 in malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:67 #1 0x738381383514 (/usr/lib/libc.so.6+0xe3514) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35) #2 0x738381383418 in fdopendir (/usr/lib/libc.so.6+0xe3418) (BuildId: 468e3585c794491a48ea75fceb9e4d6b1464fc35) #3 0x6433e1aefac4 in for_each<rgw::sal::Directory::fill_cache(const DoutPrefixProvider*, optional_yield, rgw::sal::fill_cache_cb_t&)::<lambda(char const*)> > /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:874 #4 0x6433e1aa10ab in rgw::sal::Directory::fill_cache(DoutPrefixProvider const*, optional_yield, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, int (DoutPrefixProvider const*, rgw_bucket_dir_ent ry&) const> > const&) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:1077 #5 0x6433e1aa8974 in rgw::sal::MPDirectory::fill_cache(DoutPrefixProvider const*, optional_yield, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, int (DoutPrefixProvider const*, rgw_bucket_dir_e ntry&) const> > const&) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:1384 #6 0x6433e1abf1ee in rgw::sal::POSIXBucket::fill_cache(DoutPrefixProvider const*, optional_yield, fu2::abi_310::detail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, int (DoutPrefixProvider const*, rgw_bucket_dir_e ntry&) const> > const&) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:2236 #7 0x6433e1c43956 in file::listing::BucketCache<rgw::sal::POSIXDriver, rgw::sal::POSIXBucket>::fill(DoutPrefixProvider const*, file::listing::BucketCacheEntry<rgw::sal::POSIXDriver, rgw::sal::POSIXBucket>*, rgw::sal::POSIXBucket*, unsigned int, optional_yield) /home/kef u/dev/ceph/src/rgw/driver/posix/bucket_cache.h:368 #8 0x6433e1c1c71c in file::listing::BucketCache<rgw::sal::POSIXDriver, rgw::sal::POSIXBucket>::list_bucket(DoutPrefixProvider const*, optional_yield, rgw::sal::POSIXBucket*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, fu2::abi_310::d etail::function<fu2::abi_310::detail::config<true, false, 16ul>, fu2::abi_310::detail::property<true, false, bool (rgw_bucket_dir_entry const&) const> >) /home/kefu/dev/ceph/src/rgw/driver/posix/bucket_cache.h:410 ceph#9 0x6433e1ac1829 in rgw::sal::POSIXBucket::list(DoutPrefixProvider const*, rgw::sal::Bucket::ListParams&, int, rgw::sal::Bucket::ListResults&, optional_yield) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:2256 ceph#10 0x6433e1ae1302 in rgw::sal::POSIXMultipartUpload::list_parts(DoutPrefixProvider const*, ceph::common::CephContext*, int, int, int*, bool*, optional_yield, bool) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:3692 ceph#11 0x6433e1ae2f3b in rgw::sal::POSIXMultipartUpload::complete(DoutPrefixProvider const*, optional_yield, ceph::common::CephContext*, std::map<int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<int>, std::allocator<std::pair< int const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > >&, std::__cxx11::list<rgw_obj_index_key, std::allocator<rgw_obj_index_key> >&, unsigned long&, bool&, RGWCompressionInfo&, long&, std::__cxx11::basic_string<char, std::char_trait s<char>, std::allocator<char> >&, ACLOwner&, unsigned long, rgw::sal::Object*, boost::container::flat_map<unsigned int, boost::container::flat_set<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std ::char_traits<char>, std::allocator<char> > >, void>, std::less<unsigned int>, void>&) /home/kefu/dev/ceph/src/rgw/driver/posix/rgw_sal_posix.cc:3770 ceph#12 0x6433e0fffb73 in POSIXMPObjectTest::create_MPObj(rgw::sal::MultipartUpload*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >) /home/kefu/dev/ceph/src/test/rgw/test_rgw_posix_driver.cc:1717 ceph#13 0x6433e0e95ab9 in POSIXMPObjectTest_BucketList_Test::TestBody() /home/kefu/dev/ceph/src/test/rgw/test_rgw_posix_driver.cc:1863 ceph#14 0x6433e1308d17 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 ``` Fixes https://tracker.ceph.com/issues/71505 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, SyncPoint allocated two C_Gather instances tracked by raw pointers but failed to properly clean them up when only a single sync point existed, causing memory leaks detected by AddressSanitizer. This change fixes the leak by modifying AbstractWriteLog::shut_down() to check for prior sync points in the chain. When the current sync point is the only one present, we now activate the m_prior_log_entries_persisted context to ensure: - The onfinish callback executes and releases the captured strong reference to the enclosing SyncPoint - The parent m_sync_point_persist context completes and gets properly released This ensures all allocated contexts are cleaned up correctly during shutdown, eliminating the memory leak. The ASan report: ``` Indirect leak of 2064 byte(s) in 1 object(s) allocated from: #0 0x56440919ae2d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_librbd+0x2f3de2d) (BuildId: 6a04677c6ee5235f1a41815df807f97c5b96d4cd) #1 0x56440bd67751 in __gnu_cxx::new_allocator<Context*>::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x56440bd676e0 in std::allocator<Context*>::allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x56440bd676e0 in std::allocator_traits<std::allocator<Context*>>::allocate(std::allocator<Context*>&, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20 #4 0x56440bd6730b in std::_Vector_base<Context*, std::allocator<Context*>>::_M_allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:346:20 #5 0x7fd33e00e8d1 in std::vector<Context*, std::allocator<Context*>>::reserve(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:78:22 #6 0x7fd33e00c51c in librbd::cache::pwl::SyncPoint::SyncPoint(unsigned long, ceph::common::CephContext*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/SyncPoint.cc:20:27 #7 0x56440bd65f26 in decltype(::new((void*)(0)) librbd::cache::pwl::SyncPoint(std::declval<unsigned long&>(), std::declval<ceph::common::CephContext*&>())) std::construct_at<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(librbd::cache::pwl::SyncPoint*, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 #8 0x56440bd65b98 in void std::allocator_traits<std::allocator<librbd::cache::pwl::SyncPoint>>::construct<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint>&, librbd::cache::pwl::SyncPoint*, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:518:4 ceph#9 0x56440bd657d3 in std::_Sp_counted_ptr_inplace<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:519:4 ceph#10 0x56440bd65371 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(librbd::cache::pwl::SyncPoint*&, std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:651:6 ceph#11 0x56440bd65163 in std::__shared_ptr<librbd::cache::pwl::SyncPoint, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1342:14 ceph#12 0x56440bd650e6 in std::shared_ptr<librbd::cache::pwl::SyncPoint>::shared_ptr<std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4 ceph#13 0x56440bd65057 in std::shared_ptr<librbd::cache::pwl::SyncPoint> std::allocate_shared<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint> const&, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:862:14 ceph#14 0x56440bca97e7 in std::shared_ptr<librbd::cache::pwl::SyncPoint> std::make_shared<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:878:14 ceph#15 0x56440bd443c8 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::new_sync_point(librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1905:20 ceph#16 0x56440bd42e4c in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::flush_new_sync_point(librbd::cache::pwl::C_FlushRequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1951:3 ceph#17 0x56440bd9cbf2 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::flush_new_sync_point_if_needed(librbd::cache::pwl::C_FlushRequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1990:5 ceph#18 0x56440bd9c636 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*)::'lambda'(librbd::cache::pwl::GuardedRequestFunctionContext&)::operator()(librbd::cache::pwl::GuardedRequestFunctionContext&) const /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:2152:9 ceph#19 0x56440bd9b9b4 in boost::detail::function::void_function_obj_invoker<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*)::'lambda'(librbd::cache::pwl::GuardedRequestFunctionContext&), void, librbd::cache::pwl::GuardedRequestFunctionContext&>::invoke(boost::detail::function::function_buffer&, librbd::cache::pwl::GuardedRequestFunctionContext&) /opt/ceph/include/boost/function/function_template.hpp:100:11 ceph#20 0x56440bd29321 in boost::function_n<void, librbd::cache::pwl::GuardedRequestFunctionContext&>::operator()(librbd::cache::pwl::GuardedRequestFunctionContext&) const /opt/ceph/include/boost/function/function_template.hpp:789:14 ceph#21 0x56440bd28d85 in librbd::cache::pwl::GuardedRequestFunctionContext::finish(int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/Request.h:335:5 ceph#22 0x5644091e0fe0 in Context::complete(int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/include/Context.h:102:5 ceph#23 0x56440bd9b378 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::detain_guarded_request(librbd::cache::pwl::C_BlockIORequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::GuardedRequestFunctionContext*, bool) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1202:20 ceph#24 0x56440bd96c50 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:2154:3 ceph#25 0x56440bd1e4b5 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::shut_down(Context*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:703:3 ceph#26 0x56440bdb9022 in librbd::cache::pwl::TestMockCacheSSDWriteLog_compare_and_write_compare_matched_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/cache/pwl/test_mock_SSDWriteLog.cc:403:7 ``` Fixes: https://tracker.ceph.com/issues/71335 Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, SyncPoint allocated two C_Gather instances tracked by raw pointers but failed to properly clean them up when only a single sync point existed, causing memory leaks detected by AddressSanitizer. This change fixes the leak by modifying AbstractWriteLog::shut_down() to check for prior sync points in the chain. When the current sync point is the only one present, we now activate the m_prior_log_entries_persisted context to ensure: - The onfinish callback executes and releases the captured strong reference to the enclosing SyncPoint - The parent m_sync_point_persist context completes and gets properly released This ensures all allocated contexts are cleaned up correctly during shutdown, eliminating the memory leak. The ASan report: ``` Indirect leak of 2064 byte(s) in 1 object(s) allocated from: #0 0x56440919ae2d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_librbd+0x2f3de2d) (BuildId: 6a04677c6ee5235f1a41815df807f97c5b96d4cd) #1 0x56440bd67751 in __gnu_cxx::new_allocator<Context*>::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x56440bd676e0 in std::allocator<Context*>::allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x56440bd676e0 in std::allocator_traits<std::allocator<Context*>>::allocate(std::allocator<Context*>&, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:464:20 #4 0x56440bd6730b in std::_Vector_base<Context*, std::allocator<Context*>>::_M_allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_vector.h:346:20 #5 0x7fd33e00e8d1 in std::vector<Context*, std::allocator<Context*>>::reserve(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/vector.tcc:78:22 #6 0x7fd33e00c51c in librbd::cache::pwl::SyncPoint::SyncPoint(unsigned long, ceph::common::CephContext*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/SyncPoint.cc:20:27 #7 0x56440bd65f26 in decltype(::new((void*)(0)) librbd::cache::pwl::SyncPoint(std::declval<unsigned long&>(), std::declval<ceph::common::CephContext*&>())) std::construct_at<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(librbd::cache::pwl::SyncPoint*, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/stl_construct.h:97:39 #8 0x56440bd65b98 in void std::allocator_traits<std::allocator<librbd::cache::pwl::SyncPoint>>::construct<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint>&, librbd::cache::pwl::SyncPoint*, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/alloc_traits.h:518:4 ceph#9 0x56440bd657d3 in std::_Sp_counted_ptr_inplace<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:519:4 ceph#10 0x56440bd65371 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(librbd::cache::pwl::SyncPoint*&, std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:651:6 ceph#11 0x56440bd65163 in std::__shared_ptr<librbd::cache::pwl::SyncPoint, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr_base.h:1342:14 ceph#12 0x56440bd650e6 in std::shared_ptr<librbd::cache::pwl::SyncPoint>::shared_ptr<std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::_Sp_alloc_shared_tag<std::allocator<librbd::cache::pwl::SyncPoint>>, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:409:4 ceph#13 0x56440bd65057 in std::shared_ptr<librbd::cache::pwl::SyncPoint> std::allocate_shared<librbd::cache::pwl::SyncPoint, std::allocator<librbd::cache::pwl::SyncPoint>, unsigned long&, ceph::common::CephContext*&>(std::allocator<librbd::cache::pwl::SyncPoint> const&, unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:862:14 ceph#14 0x56440bca97e7 in std::shared_ptr<librbd::cache::pwl::SyncPoint> std::make_shared<librbd::cache::pwl::SyncPoint, unsigned long&, ceph::common::CephContext*&>(unsigned long&, ceph::common::CephContext*&) /usr/lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/shared_ptr.h:878:14 ceph#15 0x56440bd443c8 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::new_sync_point(librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1905:20 ceph#16 0x56440bd42e4c in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::flush_new_sync_point(librbd::cache::pwl::C_FlushRequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1951:3 ceph#17 0x56440bd9cbf2 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::flush_new_sync_point_if_needed(librbd::cache::pwl::C_FlushRequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::DeferredContexts&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1990:5 ceph#18 0x56440bd9c636 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*)::'lambda'(librbd::cache::pwl::GuardedRequestFunctionContext&)::operator()(librbd::cache::pwl::GuardedRequestFunctionContext&) const /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:2152:9 ceph#19 0x56440bd9b9b4 in boost::detail::function::void_function_obj_invoker<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*)::'lambda'(librbd::cache::pwl::GuardedRequestFunctionContext&), void, librbd::cache::pwl::GuardedRequestFunctionContext&>::invoke(boost::detail::function::function_buffer&, librbd::cache::pwl::GuardedRequestFunctionContext&) /opt/ceph/include/boost/function/function_template.hpp:100:11 ceph#20 0x56440bd29321 in boost::function_n<void, librbd::cache::pwl::GuardedRequestFunctionContext&>::operator()(librbd::cache::pwl::GuardedRequestFunctionContext&) const /opt/ceph/include/boost/function/function_template.hpp:789:14 ceph#21 0x56440bd28d85 in librbd::cache::pwl::GuardedRequestFunctionContext::finish(int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/Request.h:335:5 ceph#22 0x5644091e0fe0 in Context::complete(int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/include/Context.h:102:5 ceph#23 0x56440bd9b378 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::detain_guarded_request(librbd::cache::pwl::C_BlockIORequest<librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>>*, librbd::cache::pwl::GuardedRequestFunctionContext*, bool) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:1202:20 ceph#24 0x56440bd96c50 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::internal_flush(bool, Context*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:2154:3 ceph#25 0x56440bd1e4b5 in librbd::cache::pwl::AbstractWriteLog<librbd::MockImageCtx>::shut_down(Context*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/librbd/cache/pwl/AbstractWriteLog.cc:703:3 ceph#26 0x56440bdb9022 in librbd::cache::pwl::TestMockCacheSSDWriteLog_compare_and_write_compare_matched_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/librbd/cache/pwl/test_mock_SSDWriteLog.cc:403:7 ``` Fixes: https://tracker.ceph.com/issues/71335 Signed-off-by: Kefu Chai <tchaikov@gmail.com> (cherry picked from commit 05fd6f9)

Removing vxattr 'ceph.dir.subvolume' on a directory without it being set causes the mds to crash. This is because the snaprealm would be null for the directory and the null check is missing. Setting the vxattr, creates the snaprealm for the directory as part of it. Hence, mds doesn't crash when the vxattr is set and then removed. This patch fixes the same. Reproducer: $mkdir /mnt/dir1 $setfattr -x "ceph.dir.subvolume" /mnt/dir1 Traceback: ------- Core was generated by `./ceph/build/bin/ceph-mds -i a -c ./ceph/build/ceph.conf'. Program terminated with signal SIGSEGV, Segmentation fault. (gdb) bt #0 0x00007f33f1aa8034 in __pthread_kill_implementation () from /lib64/libc.so.6 #1 0x00007f33f1a4ef1e in raise () from /lib64/libc.so.6 #2 0x0000562b148a6fd0 in reraise_fatal (signum=signum@entry=11) at /ceph/src/global/signal_handler.cc:88 #3 0x0000562b148a83d9 in handle_oneshot_fatal_signal (signum=11) at /ceph/src/global/signal_handler.cc:367 #4 <signal handler called> #5 Server::handle_client_setvxattr (this=0x562b4ee3f800, mdr=..., cur=0x562b4ef9cc00) at /ceph/src/mds/Server.cc:6406 #6 0x0000562b145fadc2 in Server::handle_client_removexattr (this=0x562b4ee3f800, mdr=...) at /ceph/src/mds/Server.cc:7022 #7 0x0000562b145fbff0 in Server::dispatch_client_request (this=0x562b4ee3f800, mdr=...) at /ceph/src/mds/Server.cc:2825 #8 0x0000562b145fcfa2 in Server::handle_client_request (this=0x562b4ee3f800, req=...) at /ceph/src/mds/Server.cc:2676 ceph#9 0x0000562b1460063c in Server::dispatch (this=0x562b4ee3f800, m=...) at /ceph/src/mds/Server.cc:382 ceph#10 0x0000562b1450eb22 in MDSRank::handle_message (this=this@entry=0x562b4ef42008, m=...) at /ceph/src/mds/MDSRank.cc:1222 ceph#11 0x0000562b14510c93 in MDSRank::_dispatch (this=this@entry=0x562b4ef42008, m=..., new_msg=new_msg@entry=true) at /ceph/src/mds/MDSRank.cc:1045 ceph#12 0x0000562b14511620 in MDSRankDispatcher::ms_dispatch (this=this@entry=0x562b4ef42000, m=...) at /ceph/src/mds/MDSRank.cc:1019 ceph#13 0x0000562b144ff117 in MDSDaemon::ms_dispatch2 (this=0x562b4ee64000, m=...) at /ceph/src/common/RefCountedObj.h:56 ceph#14 0x00007f33f2f4974a in Messenger::ms_deliver_dispatch (this=0x562b4ee70000, m=...) at /ceph/src/msg/Messenger.h:746 ceph#15 0x00007f33f2f467e2 in DispatchQueue::entry (this=0x562b4ee703b8) at /ceph/src/msg/DispatchQueue.cc:202 ceph#16 0x00007f33f2ff61cb in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /ceph/src/msg/DispatchQueue.h:101 ceph#17 0x00007f33f2df4b5d in Thread::entry_wrapper (this=0x562b4ee70518) at /ceph/src/common/Thread.cc:87 ceph#18 0x00007f33f2df4b6f in Thread::_entry_func (arg=<optimized out>) at /ceph/src/common/Thread.cc:74 ceph#19 0x00007f33f1aa6088 in start_thread () from /lib64/libc.so.6 ceph#20 0x00007f33f1b29f8c in clone3 () from /lib64/libc.so.6 --------- Fixes: https://tracker.ceph.com/issues/70794 Signed-off-by: Kotresh HR <khiremat@redhat.com>

Fix a memory leak where HybridAllocator's embedded BitmapAllocator (bmap_alloc) was not properly cleaned up during destruction. The issue occurred because virtual function calls in destructors don't dispatch to derived class implementations - when AvlAllocator::~AvlAllocator() calls shutdown(), it invokes AvlAllocator::shutdown() rather than HybridAllocatorBase::shutdown(), leaving bmap_alloc unfreed. This issue only affects unit tests and does not impact production, as BlueFS always calls shutdown() explicitly when stopping allocators in BlueFS::_stop_alloc() This change has minimal impact on existing code that already calls shutdown() explicitly, since shutdown() has idempotent behavior and can be safely called multiple times. Additionally, destructors are only invoked during system teardown, so the potential double shutdown() call does not affect runtime performance. Changes: - Replace raw pointer bmap_alloc with unique_ptr for automatic cleanup - Add explicit shutdown() call in BitmapAllocator destructor for RAII - Replace manual bmap_alloc->shutdown() with bmap_alloc.reset() - Add explicit default destructor to HybridAllocatorBase for clarity This eliminates the need for manual shutdown() calls and ensures proper resource cleanup through RAII principles. Fixes ASan-reported indirect leak of BitmapAllocator resources. Please see the leak reported by ASan for more details: ``` Indirect leak of 23 byte(s) in 1 object(s) allocated from: #0 0x7fe30b121a2d in operator new(unsigned long) /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_new_delete.cpp:86 #1 0x5614e057ee11 in std::__new_allocator<char>::allocate(unsigned long, void const*) /usr/include/c++/15.1.1/bits/new_allocator.h:151 #2 0x5614e057d279 in std::allocator<char>::allocate(unsigned long) /usr/include/c++/15.1.1/bits/allocator.h:203 #3 0x5614e057d279 in std::allocator_traits<std::allocator<char> >::allocate(std::allocator<char>&, unsigned long) /usr/include/c++/15.1.1/bits/alloc_traits.h:614 #4 0x5614e057d279 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_S_allocate(std::allocator<char>&, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.h:142 #5 0x5614e057cf27 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_create(unsigned long&, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.tcc:164 #6 0x5614e05fe91b in void std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_construct<true>(char const*, unsigned long) /usr/include/c++/15.1.1/bits/basic_string.tcc:291 #7 0x5614e05f3cd4 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /usr/include/c++/15.1. 1/bits/basic_string.h:617 #8 0x5614e069b892 in ceph::mutex_debug_detail::mutex_debug_impl<false>::mutex_debug_impl(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, bool) /home/kefu/dev/ceph/src/common/mutex_debug. h:103 ceph#9 0x5614e06b8586 in ceph::mutex_debug_detail::mutex_debug_impl<false> ceph::make_mutex<char const (&) [23]>(char const (&) [23]) /home/kefu/dev/ceph/src/common/ceph_mutex.h:118 ceph#10 0x5614e06b8350 in AllocatorLevel02<AllocatorLevel01Loose>::AllocatorLevel02() /home/kefu/dev/ceph/src/os/bluestore/fastbmap_allocator_impl.h:526 ceph#11 0x5614e06b393b in BitmapAllocator::BitmapAllocator(ceph::common::CephContext*, long, long, std::basic_string_view<char, std::char_traits<char> >) /home/kefu/dev/ceph/src/os/bluestore/BitmapAllocator.cc:16 ceph#12 0x5614e072755f in HybridAllocatorBase<AvlAllocator>::_spillover_range(unsigned long, unsigned long) /home/kefu/dev/ceph/src/os/bluestore/HybridAllocator_impl.h:123 ceph#13 0x5614e06c9045 in AvlAllocator::_try_insert_range(unsigned long, unsigned long, boost::intrusive::tree_iterator<boost::intrusive::mhtraits<range_seg_t, boost::intrusive::avl_set_member_hook<>, &range_seg_t::offset_hook>, false>*) /home/kefu/dev/ceph/src/os/bluestore/AvlAllocator.h:210 ceph#14 0x5614e06c0104 in AvlAllocator::_add_to_tree(unsigned long, unsigned long) /home/kefu/dev/ceph/src/os/bluestore/AvlAllocator.cc:136 ceph#15 0x5614e0586e5f in HybridAllocatorBase<AvlAllocator>::_add_to_tree(unsigned long, unsigned long) /home/kefu/dev/ceph/src/os/bluestore/HybridAllocator.h:110 ceph#16 0x5614e06c6559 in AvlAllocator::init_add_free(unsigned long, unsigned long) /home/kefu/dev/ceph/src/os/bluestore/AvlAllocator.cc:494 ceph#17 0x5614e0582943 in HybridAllocator_fragmentation_Test::TestBody() /home/kefu/dev/ceph/src/test/objectstore/hybrid_allocator_test.cc:285 ceph#18 0x5614e061bee3 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2653 ceph#19 0x5614e0606562 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/kefu/dev/ceph/src/googletest/googletest/src/gtest.cc:2689` ``` Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Previously, run-cli-tests ignored all environment variables from the parent process to ensure a clean test environment. However, this also dropped sanitizer settings (ASAN_OPTIONS and LSAN_OPTIONS) needed when AddressSanitizer is enabled. This causes test failures with TCMalloc due to false-positive leak reports from TCMalloc's internal objects, which is a known issue documented in Google's C++ style guide. While recent gperftools releases have fixed this, Ubuntu Jammy still ships with an older version that triggers these warnings. This change preserves sanitizer environment variables while maintaining the clean test environment for other variables. Note: Once we upgrade to newer gperftools, we can remove the related suppression rule in qa/lsan.supp. The test failure with TCMalloc looks like: ``` /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/ceph-kvstore-tool/help.t: failed --- /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/ceph-kvstore-tool/help.t +++ /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/cli/ceph-kvstore-tool/help.t.err @@ -21,3 +21,19 @@ stats histogram [prefix] + + ================================================================= + ==87908==ERROR: LeakSanitizer: detected memory leaks + + Direct leak of 45 byte(s) in 1 object(s) allocated from: + #0 0x562fd797265d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/ceph-kvstore-tool+0xe5e65d) (BuildId: 7eb56077b615aeb3c7aedafa0818ad89fdf3702d) + #1 0x562fd79815c8 in std::__new_allocator<char>::allocate(unsigned long, void const*) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/new_allocator.h:137:27 + #2 0x562fd7981520 in std::allocator<char>::allocate(unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/allocator.h:188:32 + #3 0x562fd7981520 in std::allocator_traits<std::allocator<char>>::allocate(std::allocator<char>&, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/alloc_traits.h:464:20 + #4 0x562fd798115a in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_create(unsigned long&, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.tcc:155:14 + #5 0x562fd798787f in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_mutate(unsigned long, unsigned long, char const*, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.tcc:328:21 + #6 0x562fd79876a7 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>::_M_append(char const*, unsigned long) /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/basic_string.tcc:420:8 + #7 0x7fa1aa0286f0 in MallocExtension::Initialize() (/lib/x86_64-linux-gnu/libtcmalloc.so.4+0x2a6f0) (BuildId: eeef3d1257388a806e122398dbce3157ee568ef4) + + SUMMARY: AddressSanitizer: 45 byte(s) leaked in 1 allocation(s). ``` Signed-off-by: Kefu Chai <tchaikov@gmail.com>

Replace unsafe string construction with bufferlist::length() to avoid reading beyond buffer boundaries. In commit 92ccbff, we introduced a bug when checking if a digest was empty by constructing a std::string from bufferlist: ``` std::string(digest.second.c_str()).empty() ``` This is unsafe because bufferlist data is not guaranteed to be null- terminated. The std::string constructor searches for a null terminator and may read beyond the bufferlist's allocated memory, causing a heap-buffer-overflow detected by AddressSanitizer: ``` ==66092==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7e0c65215004 at pc 0x7fbc6e27c597 bp 0x7ffe29fb6100 sp 0x7ffe29fb58b8 READ of size 5 at 0x7e0c65215004 thread T0 #0 0x7fbc6e27c596 in strlen /usr/src/debug/gcc/gcc/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:425 #1 0x562c75fad91a in std::char_traits<char>::length(char const*) /usr/include/c++/15.2.1/bits/char_traits.h:393 #2 0x562c75fb4222 in std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::basic_string<std::allocator<char> >(char const*, std::allocator<char> const&) /usr/include/c++/15.2.1/bits/b asic_string.h:713 #3 0x562c761b81ae in operator() /home/kefu/dev/ceph/src/osd/scrubber/scrub_backend.cc:1300 #4 0x562c761d7d53 in operator()<mini_flat_map<shard_id_t, ceph::buffer::v15_2_0::list, signed char>::_iterator<false> > /usr/include/c++/15.2.1/bits/predefined_ops.h:318 #5 0x562c761d789c in __find_if<mini_flat_map<shard_id_t, ceph::buffer::v15_2_0::list, signed char>::_iterator<false>, __gnu_cxx::__ops::_Iter_pred<ScrubBackend::match_in_shards(const hobject_t&, auth_selection_ t&, inconsistent_obj_wrapper&, std::stringstream&)::<lambda(const std::pair<const shard_id_t, ceph::buffer::v15_2_0::list&>&)> > > /usr/include/c++/15.2.1/bits/stl_algobase.h:2095 #6 0x562c761d72b2 in find_if<mini_flat_map<shard_id_t, ceph::buffer::v15_2_0::list, signed char>::_iterator<false>, ScrubBackend::match_in_shards(const hobject_t&, auth_selection_t&, inconsistent_obj_wrapper&, std::stringstream&)::<lambda(const std::pair<const shard_id_t, ceph::buffer::v15_2_0::list&>&)> > /usr/include/c++/15.2.1/bits/stl_algo.h:3921 #7 0x562c761d5f6f in none_of<mini_flat_map<shard_id_t, ceph::buffer::v15_2_0::list, signed char>::_iterator<false>, ScrubBackend::match_in_shards(const hobject_t&, auth_selection_t&, inconsistent_obj_wrapper&, std::stringstream&)::<lambda(const std::pair<const shard_id_t, ceph::buffer::v15_2_0::list&>&)> > /usr/include/c++/15.2.1/bits/stl_algo.h:431 #8 0x562c761d4a50 in any_of<mini_flat_map<shard_id_t, ceph::buffer::v15_2_0::list, signed char>::_iterator<false>, ScrubBackend::match_in_shards(const hobject_t&, auth_selection_t&, inconsistent_obj_wrapper&, s td::stringstream&)::<lambda(const std::pair<const shard_id_t, ceph::buffer::v15_2_0::list&>&)> > /usr/include/c++/15.2.1/bits/stl_algo.h:450 ceph#9 0x562c761bb84b in ScrubBackend::match_in_shards(hobject_t const&, auth_selection_t&, inconsistent_obj_wrapper&, std::__cxx11::basic_stringstream<char, std::char_traits<char>, std::allocator<char> >&) /home/k efu/dev/ceph/src/osd/scrubber/scrub_backend.cc:1297 ceph#10 0x562c761b4282 in ScrubBackend::compare_obj_in_maps[abi:cxx11](hobject_t const&) /home/kefu/dev/ceph/src/osd/scrubber/scrub_backend.cc:941 ceph#11 0x562c761d44af in operator()<hobject_t> /home/kefu/dev/ceph/src/osd/scrubber/scrub_backend.cc:887 ceph#12 0x562c761d4836 in for_each<std::_Rb_tree_const_iterator<hobject_t>, ScrubBackend::compare_smaps()::<lambda(const auto:422&)> > /usr/include/c++/15.2.1/bits/stl_algo.h:3798 ceph#13 0x562c761b3259 in ScrubBackend::compare_smaps() /home/kefu/dev/ceph/src/osd/scrubber/scrub_backend.cc:884 ceph#14 0x562c761a478d in ScrubBackend::update_authoritative() /home/kefu/dev/ceph/src/osd/scrubber/scrub_backend.cc:315` ``` Fix by using bufferlist::length() which tells if the given buffer is empty instead of converting the buffer content to a string. Signed-off-by: Kefu Chai <tchaikov@gmail.com>

…ives Add suppression rules for two categories of false positive warnings encountered during ASan-enabled testing: 1. PyModule_ExecDef memory leaks: ASan incorrectly interprets Python's module loading behavior as memory leaks when the interpreter loads extension modules. 2. __cxa_throw interception failures: ASan's interceptor cannot properly intercept exception handling when libstdc++.so is loaded after the ASan shared library, causing CHECK failures. 3. ErasureCodePluginRegistry::load: `ceph::ErasureCodePluginRegistry::load()` is known to leak, as we don't free the memory allocated by the ec plugins which are registered in the `ErasureCodePluginRegistry` singleton. this is a known issue, but since the `ErasureCodePluginRegistry` instance is a singleton. we can live with it. in this change, we add the rule to suppress the leak report from LeakSanitizer. this rule also exist in qa/valgrind.supp. All warnings are confirmed false positives that should be suppressed to reduce noise in test output. Example warnings: ``` Direct leak of 3264 byte(s) in 1 object(s) allocated from: #0 0x7f6027d20cb5 in malloc /usr/src/debug/gcc/gcc/libsanitizer/asan/asan_malloc_linux.cpp:67 #1 0x7f60277557ad (/usr/lib/libpython3.13.so.1.0+0x1557ad) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) #2 0x7f6027756067 (/usr/lib/libpython3.13.so.1.0+0x156067) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) #3 0x7f60278471a0 (/usr/lib/libpython3.13.so.1.0+0x2471a0) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) #4 0x7f602774d031 (/usr/lib/libpython3.13.so.1.0+0x14d031) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) #5 0x7b60234093bb in __Pyx_modinit_type_init_code.constprop.0 /home/kefu/dev/ceph/build/src/pybind/rados/rados.c:82066 #6 0x7b602340a826 in __pyx_pymod_exec_rados /home/kefu/dev/ceph/build/src/pybind/rados/rados.c:82755 #7 0x7f6027856777 in PyModule_ExecDef (/usr/lib/libpython3.13.so.1.0+0x256777) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) #8 0x7f602785baa3 (/usr/lib/libpython3.13.so.1.0+0x25baa3) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#9 0x7f6027793df2 (/usr/lib/libpython3.13.so.1.0+0x193df2) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#10 0x7f6027777cbe in _PyEval_EvalFrameDefault (/usr/lib/libpython3.13.so.1.0+0x177cbe) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#11 0x7f60277957de (/usr/lib/libpython3.13.so.1.0+0x1957de) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#12 0x7f60277d11b9 in PyObject_CallMethodObjArgs (/usr/lib/libpython3.13.so.1.0+0x1d11b9) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#13 0x7f60277d0ee4 in PyImport_ImportModuleLevelObject (/usr/lib/libpython3.13.so.1.0+0x1d0ee4) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#14 0x7f6027779c0c in _PyEval_EvalFrameDefault (/usr/lib/libpython3.13.so.1.0+0x179c0c) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#15 0x7f602784e2c8 in PyEval_EvalCode (/usr/lib/libpython3.13.so.1.0+0x24e2c8) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#16 0x7f602788c88b (/usr/lib/libpython3.13.so.1.0+0x28c88b) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#17 0x7f602788985c (/usr/lib/libpython3.13.so.1.0+0x28985c) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#18 0x7f6027886f57 (/usr/lib/libpython3.13.so.1.0+0x286f57) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#19 0x7f6027886211 (/usr/lib/libpython3.13.so.1.0+0x286211) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#20 0x7f6027885b82 (/usr/lib/libpython3.13.so.1.0+0x285b82) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#21 0x7f6027883e50 in Py_RunMain (/usr/lib/libpython3.13.so.1.0+0x283e50) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#22 0x7f602783bbea in Py_BytesMain (/usr/lib/libpython3.13.so.1.0+0x23bbea) (BuildId: bea05fc2c8bd66145b159f10dcd810ebe813af39) ceph#23 0x7f6027227674 (/usr/lib/libc.so.6+0x27674) (BuildId: 4fe011c94a88e8aeb6f2201b9eb369f42b4a1e9e) ceph#24 0x7f6027227728 in __libc_start_main (/usr/lib/libc.so.6+0x27728) (BuildId: 4fe011c94a88e8aeb6f2201b9eb369f42b4a1e9e) ceph#25 0x55dae17e6044 in _start (/usr/bin/python3.13+0x1044) (BuildId: 8c0dc848f5b978c56ebeb07255bb332b4b37ae4e) ``` ``` AddressSanitizer: CHECK failed: asan_interceptors.cpp:335 "((__interception::real___cxa_throw)) != (0)" (0x0, 0x0) (tid=3246455) #0 0x7f345ea81979 in CheckUnwind ../../../../src/libsanitizer/asan/asan_rtl.cpp:69 #1 0x7f345eaa790d in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) ../../../../src/libsanitizer/sanitizer_common/sanitizer_termination.cpp:86 #2 0x7f345e9e1d54 in __interceptor___cxa_throw ../../../../src/libsanitizer/asan/asan_interceptors.cpp:335 #3 0x7f345e9e1d54 in __interceptor___cxa_throw ../../../../src/libsanitizer/asan/asan_interceptors.cpp:334 #4 0x7f3458623def in void boost::throw_exception<boost::bad_lexical_cast>(boost::bad_lexical_cast const&) /opt/ceph/include/boost/throw_exception.hpp:165 #5 0x7f345997ad3b in void boost::conversion::detail::throw_bad_cast<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long>() /opt/ceph/include/boost/lexical_cast/bad_lexical_cast.hpp:93 #6 0x7f3459979d35 in unsigned long boost::lexical_cast<unsigned long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) /opt/ceph/include/boost/lexical_cast.hpp:43` ``` Signed-off-by: Kefu Chai <tchaikov@gmail.com>

The static std::map max_prio_map was defined in the osd_types.h header file, causing every translation unit that included this header to get its own copy of the variable. This led to One Definition Rule (ODR) violations where multiple instances of the same variable existed at runtime. During program cleanup, destructors for these multiple instances would attempt to free the same memory regions, resulting in segmentation faults in tcmalloc/memory allocator as seen with ceph-dencoder. This issue surfaced after a yet-merged-change which converts erasure_code and json_spirit to OBJECT libraries. Before that change, these were STATIC libraries that were linked via target_link_libraries. The incorrect linkage meant their object files (and thus their copies of max_prio_map) were kept separate and didn't conflict at runtime. After converting to OBJECT libraries and properly incorporating them into libceph-common.so (commit 8b0e3fb2c23), the multiple copies of max_prio_map from different translation units all ended up in the same shared library, exposing the ODR violation. During program exit, the dynamic linker attempted to run destructors for all instances, leading to double-free crashes. Fix by moving the map into a static helper function in PeeringState.cc (the only file that uses it). The map is now a function-local static const variable, ensuring a single instance that is properly initialized and destructed. Backtrace before fix: ``` #0 0x00007ffff7dbb1a0 in tcmalloc::ThreadCache::ReleaseToCentralCache(tcmalloc::ThreadCache::FreeList*, unsigned int, int) () from /lib/x86_64-linux-gnu/libtcmalloc.so.4 #1 0x00007ffff7dbb57f in tcmalloc::ThreadCache::Scavenge() () from /lib/x86_64-linux-gnu/libtcmalloc.so.4 #2 0x00007ffff6bc8aa2 in std::__new_allocator<std::_Rb_tree_node<std::pair<int const, int> > >::deallocate (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890, __n=1) #3 0x00007ffff6bc89f9 in std::allocator<std::_Rb_tree_node<std::pair<int const, int> > >::deallocate (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890, __n=1) #4 std::allocator_traits<std::allocator<std::_Rb_tree_node<std::pair<int const, int> > > >::deallocate (__a=..., __p=0x555555f43890, __n=1) #5 std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_put_node (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890) #6 0x00007ffff6bc892e in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_drop_node (this=0x7ffff7d48f78 <max_prio_map>, __p=0x555555f43890) #7 0x00007ffff6bc886e in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43890) #8 0x00007ffff6bc8854 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43cb0) ceph#9 0x00007ffff6bc8854 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::_M_erase (this=0x7ffff7d48f78 <max_prio_map>, __x=0x555555f43ad0) ceph#10 0x00007ffff6bc8805 in std::_Rb_tree<int, std::pair<int const, int>, std::_Select1st<std::pair<int const, int> >, std::less<int>, std::allocator<std::pair<int const, int> > >::~_Rb_tree (this=0x7ffff7d48f78 <max_prio_map>) ceph#11 0x00007ffff6bc7345 in std::map<int, int, std::less<int>, std::allocator<std::pair<int const, int> > >::~map (this=0x7ffff7d48f78 <max_prio_map>) ceph#12 0x00007ffff484bd51 in __cxa_finalize (d=0x7ffff7d3f440) at ./stdlib/cxa_finalize.c:97 ceph#13 0x00007ffff6af9487 in __do_global_dtors_aux () from /home/kefu/dev/ceph/build/lib/libceph-common.so.2 ceph#14 0x00007ffff7fbfd20 in ?? () ceph#15 0x00007ffff7fc8fc2 in _dl_call_fini (closure_map=0x7fffffffd0f0, closure_map@entry=0x7ffff7fbfd20) at ./elf/dl-call_fini.c:43 ceph#16 0x00007ffff7fcbe72 in _dl_fini () at ./elf/dl-fini.c:120 ceph#17 0x00007ffff484c291 in __run_exit_handlers (status=0, listp=0x7ffff49f1680 <__exit_funcs>, run_list_atexit=run_list_atexit@entry=true, run_dtors=run_dtors@entry=true) at ./stdlib/exit.c:118 ceph#18 0x00007ffff484c35a in __GI_exit (status=<optimized out>) at ./stdlib/exit.c:148 ceph#19 0x00007ffff4833caf in __libc_start_call_main (main=main@entry=0x55555556cd90 <main(int, char const**)>, argc=argc@entry=2, argv=argv@entry=0x7fffffffd488) at ../sysdeps/nptl/libc_start_call_main.h:74 ceph#20 0x00007ffff4833d65 in __libc_start_main_impl (main=0x55555556cd90 <main(int, char const**)>, argc=2, argv=0x7fffffffd488, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7fffffffd478) at ../csu/libc-start.c:360 ceph#21 0x00005555555695e1 in _start () ``` Signed-off-by: Kefu Chai <k.chai@proxmox.com>

rpavani1998 added 2 commits May 21, 2018 22:26

Added cephfs_shell.py

416364b

Added shell.rst

0654deb

batrick changed the base branch from master to cephfs_shell May 22, 2018 00:06

batrick reviewed May 22, 2018

View reviewed changes

doc/cephfs/shell.rst

Options:

-s List file size

-S Sort by file size

Copy link

Owner

batrick May 22, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

and -l!

batrick reviewed May 22, 2018

View reviewed changes

Updated cephfs_shell.py

71c71ea

batrick closed this May 23, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

tools/cephfs: Creation of a CephFS Shell#5

tools/cephfs: Creation of a CephFS Shell#5
rpavani1998 wants to merge 3 commits intobatrick:cephfs_shellfrom
rpavani1998:cephfs_shell

rpavani1998 commented May 21, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

rpavani1998 May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

rpavani1998 May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

rpavani1998 May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

batrick May 22, 2018

Uh oh!

batrick commented May 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		-p Creates parent directories if doesn't exist.


		copyFromLocal


		Copy a file to Ceph File System from Local Directory.

		Usage : copyFromLocal [-option] <path_uri>


		Open a specific named directory.

		Usage : opendir <directory_name>

Conversation

rpavani1998 commented May 21, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batrick commented May 24, 2018

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants