Closed
Conversation
add usage for upstart osd and mon
Member
|
Incorporated this method into chef-3, along with an abstract '-all' task to allow stopping of the whole group. Thanks! |
liewegas
pushed a commit
that referenced
this pull request
Nov 18, 2012
Before the mon, and lockdep, in particular. #0 __pthread_mutex_lock (mutex=0x30) at pthread_mutex_lock.c:50 #1 0x0000000000816092 in ceph::log::Log::submit_entry (this=0x0, e=0x2f4a270) at log/Log.cc:138 #2 0x00000000007ee0f8 in handle_fatal_signal (signum=11) at global/signal_handler.cc:100 #3 <signal handler called> #4 0x00000000008e1300 in lockdep_will_lock (name=0x959aa7 "SignalHandler::lock", id=17) at common/lockdep.cc:163 #5 0x00000000008867fc in Mutex::_will_lock (this=0x2f20428) at ./common/Mutex.h:56 #6 0x0000000000886605 in Mutex::Lock (this=0x2f20428, no_lockdep=false) at common/Mutex.cc:81 #7 0x00000000007eeb95 in SignalHandler::entry (this=0x2f20300) at global/signal_handler.cc:198 #8 0x00000000008b0bd1 in Thread::_entry_func (arg=0x2f20300) at common/Thread.cc:43 #9 0x00007f36fefd6b50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #10 0x00007f36fd80b6dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #11 0x0000000000000000 in ?? () #0 0x00007f36fefd7e75 in pthread_join (threadid=139874129766144, thread_return=0x0) at pthread_join.c:89 #1 0x00000000008b11ec in Thread::join (this=0x2f20300, prval=0x0) at common/Thread.cc:130 #2 0x00000000007eeae7 in SignalHandler::shutdown (this=0x2f20300) at global/signal_handler.cc:186 #3 0x00000000007ee9cf in SignalHandler::~SignalHandler (this=0x2f20300, __in_chrg=<optimized out>) at global/signal_handler.cc:175 #4 0x00000000007eea58 in SignalHandler::~SignalHandler (this=0x2f20300, __in_chrg=<optimized out>) at global/signal_handler.cc:176 #5 0x00000000007ee643 in shutdown_async_signal_handler () at global/signal_handler.cc:324 #6 0x00000000006de9d2 in main (argc=7, argv=0x7fffbfb8a1e8) at ceph_mon.cc:439 Signed-off-by: Sage Weil <sage@inktank.com>
chamdoo
pushed a commit
to chamdoo/ceph
that referenced
this pull request
Nov 13, 2015
…ocks. Summary: SizeBeingCompacted was called without any lock protection. This causes crashes, especially when running db_bench with value_size=128K. The fix is to compute SizeUnderCompaction while holding the mutex and passing in these values into the call to Finalize. (gdb) where ceph#4 leveldb::VersionSet::SizeBeingCompacted (this=this@entry=0x7f0b490931c0, level=level@entry=4) at db/version_set.cc:1827 ceph#5 0x000000000043a3c8 in leveldb::VersionSet::Finalize (this=this@entry=0x7f0b490931c0, v=v@entry=0x7f0b3b86b480) at db/version_set.cc:1420 ceph#6 0x00000000004418d1 in leveldb::VersionSet::LogAndApply (this=0x7f0b490931c0, edit=0x7f0b3dc8c200, mu=0x7f0b490835b0, new_descriptor_log=<optimized out>) at db/version_set.cc:1016 ceph#7 0x00000000004222b2 in leveldb::DBImpl::InstallCompactionResults (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1473 ceph#8 0x0000000000426027 in leveldb::DBImpl::DoCompactionWork (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1757 ceph#9 0x0000000000426690 in leveldb::DBImpl::BackgroundCompaction (this=this@entry=0x7f0b49083400, madeProgress=madeProgress@entry=0x7f0b41bf2d1e, deletion_state=...) at db/db_impl.cc:1268 ceph#10 0x0000000000428f42 in leveldb::DBImpl::BackgroundCall (this=0x7f0b49083400) at db/db_impl.cc:1170 ceph#11 0x000000000045348e in BGThread (this=0x7f0b49023100) at util/env_posix.cc:941 ceph#12 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f0b49023100) at util/env_posix.cc:874 ceph#13 0x00007f0b4a7cf10d in start_thread (arg=0x7f0b41bf3700) at pthread_create.c:301 ceph#14 0x00007f0b49b4b11d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Test Plan: make check I am running db_bench with a value size of 128K to see if the segfault is fixed. Reviewers: MarkCallaghan, sheki, emayanke Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D9279
mathslinux
added a commit
to mathslinux/ceph
that referenced
this pull request
Mar 21, 2016
…issue-in-round-to-funcs utime.h: fix timezone issue in round_to_* funcs.
runsisi
pushed a commit
to runsisi/ceph
that referenced
this pull request
Oct 24, 2016
…er instance the caller needs to check the nullity of the parameter before calling PK11_FreeSymKey or PK11_FreeSlot, otherwise if CryptoAESKeyHandler::init failed, we will hit a segfault as follows: #0 0x00007f76844f5a95 in PK11_FreeSymKey () from /lib64/libnss3.so ceph#1 0x00007f76586b6e49 in CryptoAESKeyHandler::~CryptoAESKeyHandler() () from /lib64/librados.so.2 ceph#2 0x00007f76586b5eea in CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&) () from /lib64/librados.so.2 ceph#3 0x00007f76586b4b9c in CryptoKey::_set_secret(int, ceph::buffer::ptr const&) () from /lib64/librados.so.2 ceph#4 0x00007f76586b4e95 in CryptoKey::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#5 0x00007f76586b7ee6 in KeyRing::set_modifier(char const*, char const*, EntityName&, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&) () from /lib64/librados.so.2 ceph#6 0x00007f76586b8882 in KeyRing::decode_plaintext(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#7 0x00007f76586b9803 in KeyRing::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#8 0x00007f76586b9a1f in KeyRing::load(CephContext*, std::string const&) () from /lib64/librados.so.2 ceph#9 0x00007f76586ba04b in KeyRing::from_ceph_context(CephContext*) () from /lib64/librados.so.2 ceph#10 0x00007f765852d0cd in MonClient::init() () from /lib64/librados.so.2 ceph#11 0x00007f76583c15f5 in librados::RadosClient::connect() () from /lib64/librados.so.2 ceph#12 0x00007f765838cb1c in rados_connect () from /lib64/librados.so.2 ... Signed-off-by: runsisi <runsisi@zte.com.cn>
tchaikov
pushed a commit
that referenced
this pull request
Oct 28, 2016
…er instance the caller needs to check the nullity of the parameter before calling PK11_FreeSymKey or PK11_FreeSlot, otherwise if CryptoAESKeyHandler::init failed, we will hit a segfault as follows: #0 0x00007f76844f5a95 in PK11_FreeSymKey () from /lib64/libnss3.so #1 0x00007f76586b6e49 in CryptoAESKeyHandler::~CryptoAESKeyHandler() () from /lib64/librados.so.2 #2 0x00007f76586b5eea in CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&) () from /lib64/librados.so.2 #3 0x00007f76586b4b9c in CryptoKey::_set_secret(int, ceph::buffer::ptr const&) () from /lib64/librados.so.2 #4 0x00007f76586b4e95 in CryptoKey::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #5 0x00007f76586b7ee6 in KeyRing::set_modifier(char const*, char const*, EntityName&, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&) () from /lib64/librados.so.2 #6 0x00007f76586b8882 in KeyRing::decode_plaintext(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #7 0x00007f76586b9803 in KeyRing::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #8 0x00007f76586b9a1f in KeyRing::load(CephContext*, std::string const&) () from /lib64/librados.so.2 #9 0x00007f76586ba04b in KeyRing::from_ceph_context(CephContext*) () from /lib64/librados.so.2 #10 0x00007f765852d0cd in MonClient::init() () from /lib64/librados.so.2 #11 0x00007f76583c15f5 in librados::RadosClient::connect() () from /lib64/librados.so.2 #12 0x00007f765838cb1c in rados_connect () from /lib64/librados.so.2 ... Signed-off-by: runsisi <runsisi@zte.com.cn>
liewegas
pushed a commit
that referenced
this pull request
Dec 14, 2016
suites/fs/basic: add mds_creation_retry Barely- Reviewed-by: Greg Farnum <greg@inktank.com>
4 tasks
chardan
pushed a commit
to chardan/ceph
that referenced
this pull request
Jun 6, 2017
Abhishekvrshny
pushed a commit
to Abhishekvrshny/ceph
that referenced
this pull request
Jul 13, 2017
sebastian-philipp
pushed a commit
to sebastian-philipp/ceph
that referenced
this pull request
Jan 24, 2018
mgr/dashboard_v2: Linting support and tox configuration
liewegas
added a commit
that referenced
this pull request
May 1, 2018
The path #9 Objecter::_finish_command (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10, r=<optimized out>, rs="osd down") at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:4950 #10 0x00007f76d26de106 in Objecter::_check_command_map_dne (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1726 #11 0x00007f76d26e52e4 in Objecter::_scan_requests (this=this@entry=0x7f76c00aeb30, s=0x7f76c00af8a0, skipped_map=skipped_map@entry=false, cluster_full=cluster_full@entry=false, pool_full_map=0x7f76be7fb330, need_resend=..., need_resend_linger=..., need_resend_command=std::map with 0 elements, sul=..., gap_removed_snaps=0x7f76ac0016f8) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1120 #12 0x00007f76d26eded5 in Objecter::handle_osd_map (this=this@entry=0x7f76c00aeb30, m=m@entry=0x7f76ac0014a0) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1228 led to recursive lock of the session mutex (locked in _scan_requests, and again in _finish_command). Fix by making the callers for _finish_command (and _check_command_map_dne) take the session lock. Fixes: http://tracker.ceph.com/issues/23940 Signed-off-by: Sage Weil <sage@redhat.com>
pdvian
referenced
this pull request
in pdvian/ceph
May 11, 2018
The path #9 Objecter::_finish_command (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10, r=<optimized out>, rs="osd down") at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:4950 #10 0x00007f76d26de106 in Objecter::_check_command_map_dne (this=this@entry=0x7f76c00aeb30, c=c@entry=0x7f76b0000b10) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1726 #11 0x00007f76d26e52e4 in Objecter::_scan_requests (this=this@entry=0x7f76c00aeb30, s=0x7f76c00af8a0, skipped_map=skipped_map@entry=false, cluster_full=cluster_full@entry=false, pool_full_map=0x7f76be7fb330, need_resend=..., need_resend_linger=..., need_resend_command=std::map with 0 elements, sul=..., gap_removed_snaps=0x7f76ac0016f8) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1120 #12 0x00007f76d26eded5 in Objecter::handle_osd_map (this=this@entry=0x7f76c00aeb30, m=m@entry=0x7f76ac0014a0) at /build/ceph-13.0.2-1932-g458b4fb/src/osdc/Objecter.cc:1228 led to recursive lock of the session mutex (locked in _scan_requests, and again in _finish_command). Fix by making the callers for _finish_command (and _check_command_map_dne) take the session lock. Fixes: http://tracker.ceph.com/issues/23940 Signed-off-by: Sage Weil <sage@redhat.com> (cherry picked from commit 891f519)
dotnwat
added a commit
that referenced
this pull request
Jul 29, 2018
style: run gofmt on all sources Signed-off-by: Noah Watkins <noahwatkins@gmail.com>
dotnwat
pushed a commit
that referenced
this pull request
Jul 29, 2018
…nCommand(), which panics Go 1.6. Addresses issue #10.
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 13, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 17, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 21, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 22, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 22, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Oct 25, 2018
``` Thread 83 "ms_dispatch" hit Hardware watchpoint 4: *0x555560aaa03a Old value = 538 New value = 0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 136 ../sysdeps/x86_64/multiarch/memset-avx2.S: No such file or directory. (gdb) bt #0 __memset_avx2 () at ../sysdeps/x86_64/multiarch/memset-avx2.S:136 #1 0x00005555562e0383 in memset (__len=3934, __ch=0, __dest=<optimized out>) at /usr/include/x86_64-linux-gnu/bits/string3.h:90 #2 ceph::buffer::ptr::append_zeros (this=this@entry=0x5555608db8a0, l=l@entry=3934) at /work/ceph-4/src/common/buffer.cc:780 ceph#3 0x00005555562e426d in ceph::buffer::list::append_zero (this=0x7fffe2db22b0, len=<optimized out>) at /work/ceph-4/src/common/buffer.cc:1740 ceph#4 0x0000555555f47e29 in BlueStore::_apply_padding (this=0x55556059c000, head_pad=0, tail_pad=4038, padded=...) at /work/ceph-4/src/os/bluestore/BlueStore.cc:12595 ceph#5 0x0000555555f49414 in BlueStore::_do_write_small (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, blp=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10560 ceph#6 0x0000555555f4b4ba in BlueStore::_do_write_data (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., wctx=0x7fffe2db24e0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11157 ceph#7 0x0000555555f4c0e7 in BlueStore::_do_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11375 ceph#8 0x0000555555f4ce5b in BlueStore::_write (this=0x55556059c000, txc=0x555560779200, c=..., o=..., offset=0, length=58, bl=..., fadvise_flags=0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:11436 ceph#9 0x0000555555f513e9 in BlueStore::_txc_add_transaction (this=<optimized out>, txc=<optimized out>, t=<optimized out>) at /work/ceph-4/src/os/bluestore/BlueStore.cc:10047 ceph#10 0x0000555555f54b26 in BlueStore::queue_transactions (this=0x55556059c000, ch=..., tls=std::vector of length 1, capacity 1 = {...}, op=..., handle=0x0) at /work/ceph-4/src/os/bluestore/BlueStore.cc:9824 ceph#11 0x0000555555b2f932 in ObjectStore::queue_transaction(boost::intrusive_ptr<ObjectStore::CollectionImpl>&, ObjectStore::Transaction&&, boost::intrusive_ptr<TrackedOp>, ThreadPool::TPHandle*) ( this=0x55556059c000, ch=..., t=<optimized out>, op=..., handle=0x0) at /work/ceph-4/src/os/ObjectStore.h:1491 ceph#12 0x0000555555ad07a8 in OSD::handle_osd_map (this=0x555560744000, m=<optimized out>) at /work/ceph-4/src/osd/OSD.cc:7640 ceph#13 0x0000555555aded81 in OSD::_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6876 ceph#14 0x0000555555adf128 in OSD::ms_dispatch (this=0x555560744000, m=0x55555f74ec00) at /work/ceph-4/src/osd/OSD.cc:6555 ceph#15 0x000055555640a45a in Dispatcher::ms_dispatch2 (m=..., this=0x555560744000) at /work/ceph-4/src/msg/Dispatcher.h:125 ceph#16 Messenger::ms_deliver_dispatch (m=..., this=0x5555605c9800) at /work/ceph-4/src/msg/Messenger.h:642 ceph#17 DispatchQueue::entry (this=0x5555605c9a10) at /work/ceph-4/src/msg/DispatchQueue.cc:196 ceph#18 0x00005555562a61bd in DispatchQueue::DispatchThread::entry (this=<optimized out>) at /work/ceph-4/src/msg/DispatchQueue.h:102 ceph#19 0x00007ffff58476ba in start_thread (arg=0x7fffe2db7700) at pthread_create.c:333 ceph#20 0x00007ffff48af41d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:109 ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Mar 7, 2022
The problem is:
```
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0
DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head
=================================================================
==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8
WRITE of size 24 at 0x7f6de5176e70 thread T0
#0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156)
#1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3)
#2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c)
Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
ceph#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170
ceph#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265
ceph#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143
ceph#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206
ceph#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395
ceph#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418
ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227
ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106
ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268
ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280
ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230
ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373
ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220
ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577
ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449
ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216
ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576
Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35
ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372
ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39)
0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000)
allocated by thread T0 here:
#0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7)
#1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196
#2 0x7fff3214bc4f ([stack]+0xa5c4f)
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
zhscn
referenced
this pull request
in zhscn/ceph
Mar 14, 2022
The problem is:
```
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0
DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head
=================================================================
==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8
WRITE of size 24 at 0x7f6de5176e70 thread T0
#0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156)
#1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3)
#2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c)
Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170
#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265
#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143
#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206
#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395
#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418
#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227
#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106
#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268
ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280
ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230
ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373
ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220
ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577
ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449
ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216
ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576
Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35
ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372
ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39)
0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000)
allocated by thread T0 here:
#0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7)
#1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196
#2 0x7fff3214bc4f ([stack]+0xa5c4f)
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
nSedrickm
referenced
this pull request
in nSedrickm/ceph
Mar 21, 2022
The problem is:
```
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0
DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head
=================================================================
==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8
WRITE of size 24 at 0x7f6de5176e70 thread T0
#0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156)
#1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3)
#2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c)
Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170
#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265
#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143
#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206
#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395
#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418
#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227
#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106
#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268
#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280
#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230
#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373
#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220
#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577
#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449
#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216
#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576
Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35
#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372
#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39)
0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000)
allocated by thread T0 here:
#0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7)
#1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196
#2 0x7fff3214bc4f ([stack]+0xa5c4f)
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
This was referenced Mar 23, 2022
dpaganel
pushed a commit
to dpaganel/ceph
that referenced
this pull request
May 17, 2022
The problem is:
```
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0
DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1
DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head
=================================================================
==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8
WRITE of size 24 at 0x7f6de5176e70 thread T0
#0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156)
ceph#1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3)
ceph#2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c)
Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
ceph#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170
ceph#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265
ceph#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143
ceph#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206
ceph#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395
ceph#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418
ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)ceph#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227
ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)ceph#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106
ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268
ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280
ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230
ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373
ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220
ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577
ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449
ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216
ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576
Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39
ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35
ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372
ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39)
0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000)
allocated by thread T0 here:
#0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7)
ceph#1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196
ceph#2 0x7fff3214bc4f ([stack]+0xa5c4f)
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
m-ildefons
pushed a commit
to m-ildefons/ceph
that referenced
this pull request
Jun 15, 2022
ci: build radosgw binary
rzarzynski
added a commit
to rzarzynski/ceph
that referenced
this pull request
Jul 5, 2022
Before the patch there was a possibility that `OSDConnectionPriv`
gets destructed before a `PipelineHandle` instance that was using
it. The reason is our remote-handling operations store `conn` directly
while `handle` is defined in a parent class. Due to the language rules
the former gets deinitialized earlier.
```
==756032==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000039684 at pc 0x0000020bdfa2 bp 0x7ffd3abfa370 sp 0x7ffd3abfa360
READ of size 1 at 0x615000039684 thread T0
Reactor stalled for 261 ms on shard 0. Backtrace: 0x45d9d 0xe90f6d1 0xe6b8a1d 0xe6d1205 0xe6d16a8 0xe6d1938 0xe6d1c03 0x12cdf 0xccebf 0x7f6447161b1e 0x7f644714aee8 0x7f644714eed6 0x7f644714fb36 0x7f64471420b5 0x
7f6447143f3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0xbdc1a 0x20bdfa1 0x20c184e 0x352eb7f 0x352fa28 0x20b04a5 0x1be30e5 0xe694bc4 0xe6ebb8a 0xe843a11 0xe845a22 0xe29f497 0xe2a3ccd 0x1ab1841 0x3aca2 0x175698d
#0 0x20bdfa1 in seastar::shared_mutex::unlock() ../src/seastar/include/seastar/core/shared_mutex.hh:122
#1 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::exit() ../src/crimson/common/operation.h:548
#2 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::exit() ../src/crimson/common/operation.h:533
ceph#3 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::cancel() ../src/crimson/common/operation.h:539
ceph#4 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:543
ceph#5 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:544
ceph#6 0x352eb7f in std::default_delete<crimson::PipelineExitBarrierI>::operator()(crimson::PipelineExitBarrierI*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85
ceph#7 0x352eb7f in std::unique_ptr<crimson::PipelineExitBarrierI, std::default_delete<crimson::PipelineExitBarrierI> >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361
ceph#8 0x352eb7f in crimson::PipelineHandle::~PipelineHandle() ../src/crimson/common/operation.h:457
ceph#9 0x352eb7f in crimson::osd::PhasedOperationT<crimson::osd::ClientRequest>::~PhasedOperationT() ../src/crimson/osd/osd_operation.h:152
ceph#10 0x352eb7f in crimson::osd::ClientRequest::~ClientRequest() ../src/crimson/osd/osd_operations/client_request.cc:64
ceph#11 ...
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
3 tasks
Pegonzal
pushed a commit
that referenced
this pull request
Oct 13, 2022
Before the patch there was a possibility that `OSDConnectionPriv`
gets destructed before a `PipelineHandle` instance that was using
it. The reason is our remote-handling operations store `conn` directly
while `handle` is defined in a parent class. Due to the language rules
the former gets deinitialized earlier.
```
==756032==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000039684 at pc 0x0000020bdfa2 bp 0x7ffd3abfa370 sp 0x7ffd3abfa360
READ of size 1 at 0x615000039684 thread T0
Reactor stalled for 261 ms on shard 0. Backtrace: 0x45d9d 0xe90f6d1 0xe6b8a1d 0xe6d1205 0xe6d16a8 0xe6d1938 0xe6d1c03 0x12cdf 0xccebf 0x7f6447161b1e 0x7f644714aee8 0x7f644714eed6 0x7f644714fb36 0x7f64471420b5 0x
7f6447143f3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0xbdc1a 0x20bdfa1 0x20c184e 0x352eb7f 0x352fa28 0x20b04a5 0x1be30e5 0xe694bc4 0xe6ebb8a 0xe843a11 0xe845a22 0xe29f497 0xe2a3ccd 0x1ab1841 0x3aca2 0x175698d
#0 0x20bdfa1 in seastar::shared_mutex::unlock() ../src/seastar/include/seastar/core/shared_mutex.hh:122
#1 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::exit() ../src/crimson/common/operation.h:548
#2 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::exit() ../src/crimson/common/operation.h:533
#3 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::cancel() ../src/crimson/common/operation.h:539
#4 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:543
#5 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:544
#6 0x352eb7f in std::default_delete<crimson::PipelineExitBarrierI>::operator()(crimson::PipelineExitBarrierI*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85
#7 0x352eb7f in std::unique_ptr<crimson::PipelineExitBarrierI, std::default_delete<crimson::PipelineExitBarrierI> >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361
#8 0x352eb7f in crimson::PipelineHandle::~PipelineHandle() ../src/crimson/common/operation.h:457
#9 0x352eb7f in crimson::osd::PhasedOperationT<crimson::osd::ClientRequest>::~PhasedOperationT() ../src/crimson/osd/osd_operation.h:152
#10 0x352eb7f in crimson::osd::ClientRequest::~ClientRequest() ../src/crimson/osd/osd_operations/client_request.cc:64
#11 ...
```
Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>
tobias-urdin
pushed a commit
to tobias-urdin/ceph
that referenced
this pull request
Aug 2, 2023
c4d30d7 Ravindra Choudhari Mon, 27 Jun 2022 removing region name 4a13f58 Ravindra Choudhari Thu, 16 Jun 2022 Updating readme file (ceph#15) 18bc152 Ravindra Choudhari Tue, 14 Jun 2022 Adding attr test_of_iam to all user policy tests (ceph#13) 03f520a Ravindra Choudhari Tue, 14 Jun 2022 resolving review comments (ceph#12) 7cf2823 Ravindra Choudhari Mon, 13 Jun 2022 added IAM policy test section in README.rst (ceph#11) 563f3ea Ravindra Choudhari Fri, 10 Jun 2022 adding failing three tests back with attr @fails_on_rgw (ceph#10) 696dd2e Ravindra Choudhari Mon, 6 Jun 2022 changes as per review comments 3d63dfd Ravindra Choudhari Mon, 6 Jun 2022 Fixed review comments (ceph#8) 9492f69 Ravindra Choudhari Fri, 3 Jun 2022 Fixed review comments (ceph#7) 74095dc Ketan Arlulkar Wed, 1 Jun 2022 Fixed review comments (ceph#6) 942fb4f Ketan Arlulkar Wed, 1 Jun 2022 Added Tests for conflicting policies and IAM actions (ceph#4) ad5b5ae Ravindra Choudhari Tue, 31 May 2022 IAM policies s3 actions (ceph#5) 6515ec6 Ketan Arlulkar Fri, 27 May 2022 Corrected eq import 40a2841 Ravindra Choudhari Tue, 17 May 2022 resolving conflicts f53a5c1 Ravindra Choudhari Tue, 17 May 2022 added cleanup 747d563 Ketan Arlulkar Tue, 17 May 2022 Added cleanup/Delete Policy d1cc1d8 Ketan Arlulkar Mon, 16 May 2022 Fixed review comments 1ec43a2 Ravindra Choudhari Mon, 16 May 2022 delete user policy tests a01722e Ravindra Choudhari Mon, 16 May 2022 get user policy tests ff9d676 Ketan Arlulkar Fri, 13 May 2022 Removed TEST IDs d261400 Ketan Arlulkar Tue, 10 May 2022 Put User Policy & List User Policy Tests Signed-off-by: Ravindra Choudhari <ravindra.choudhari@seagate.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 8, 2023
Beacuse the loop's returned future is ignored,
we should cover the scenario where the pg is removed and the
snap_trimq iteration didn't complete yet.
Spotted in testing:
```
=================================================================
==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0
READ of size 8 at 0x625000f326d8 thread T0
Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
#0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172)
#1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708)
#2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf)
ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc)
ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198)
ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046)
ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196)
ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe)
ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc)
ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050)
ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1)
ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03)
ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b)
ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45)
ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac)
ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712)
ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b)
ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c)
ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e)
ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281)
ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2)
ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb)
ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8)
ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f)
ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44)
0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8)
freed by thread T0 here:
#0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b)
previously allocated by thread T0 here:
#0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82
#1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6)
#2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c)
ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87)
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&)
```
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 8, 2023
Beacuse the loop's returned future is ignored,
we should cover the scenario where the pg is removed and the
snap_trimq iteration didn't complete yet.
Spotted in testing:
```
=================================================================
==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0
READ of size 8 at 0x625000f326d8 thread T0
Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
#0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172)
#1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708)
#2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf)
ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc)
ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198)
ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046)
ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196)
ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe)
ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc)
ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050)
ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1)
ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03)
ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b)
ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45)
ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac)
ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712)
ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b)
ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c)
ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e)
ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281)
ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2)
ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb)
ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8)
ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f)
ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44)
0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8)
freed by thread T0 here:
#0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b)
previously allocated by thread T0 here:
#0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82
#1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6)
#2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c)
ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87)
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&)
```
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 8, 2023
Beacuse the loop's returned future is ignored,
we should cover the scenario where the pg is removed and the
snap_trimq iteration didn't complete yet.
Spotted in testing:
```
=================================================================
==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0
READ of size 8 at 0x625000f326d8 thread T0
Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack:
#0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172)
#1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708)
#2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf)
ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc)
ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198)
ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046)
ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196)
ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe)
ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc)
ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050)
ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1)
ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03)
ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b)
ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45)
ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac)
ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712)
ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b)
ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c)
ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e)
ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281)
ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2)
ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb)
ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8)
ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf)
ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f)
ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44)
0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8)
freed by thread T0 here:
#0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b)
previously allocated by thread T0 here:
#0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44
kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82
#1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6)
#2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c)
ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87)
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&)
```
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 9, 2023
Beacuse the loop's returned future is ignored, we should cover the scenario where the pg is removed and the snap_trimq iteration didn't complete yet. Fixes: https://tracker.ceph.com/issues/61653 Spotted in testing: ``` ================================================================= ==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0 READ of size 8 at 0x625000f326d8 thread T0 Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: #0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172) #1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708) #2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf) ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc) ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198) ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046) ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196) ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe) ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc) ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050) ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1) ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03) ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b) ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45) ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac) ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712) ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b) ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c) ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e) ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281) ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2) ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb) ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8) ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f) ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44) 0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8) freed by thread T0 here: #0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf) #1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b) previously allocated by thread T0 here: #0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367) Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82 #1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6) #2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c) ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87) SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) ``` Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 15, 2023
```
DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - ShardServices::dispatch_context_transaction: empty transaction
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - do_peering_event ignoring epoch_sent: 37 epoch_requested: 37 RenewLease -- pg has reset
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - peering_event(id=33559971, detail=PeeringEvent(from=0 pgid=42.9 sent=37 requested=37 evt=epoch_sent: 37 epoch_requested: 37 RenewLease)): submitting ctx
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - ShardServices::dispatch_context_transaction: empty transaction
#0 0x5653c613c071 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1ed27071)
DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - snaptrim_event(id=10610, detail=SnapTrimEvent(pgid=16.1a snapid=a needs_pause=0)): interrupted crimson::common::actingset_changed (acting set changed)
DEBUG 2023-11-14 15:23:50,927 [shard 0] osd - pg_advance_map(id=10742, detail=PGAdvanceMap(pg=24.2 from=46 to=48)): complete
#1 0x5653c8670acf in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x2125bacf)
#2 0x5653c8670e22 in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ (/usr/bin/ceph-osd+0x2125be22)
ceph#3 0x5653c5d0b4b2 in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) (/usr/bin/ceph-osd+0x1e8f64b2)
ceph#4 0x5653c5d3069f in void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) (/usr/bin/ceph-osd+0x1e91b69f)
ceph#5 0x5653c5d4e09d in seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x1e93909d)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map
ceph#6 0x5653d3bc3e2b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c7aee2b)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map : Not dirtying info: last_persisted is 28 while current is 48
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - log is not dirty
ceph#7 0x5653d3c61c42 in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c84cc42)
ceph#8 0x5653d3e84afa in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2ca6fafa)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10706, detail=PGAdvanceMap(pg=17.17 from=46 to=48)): exit
ceph#9 0x5653d3e88306 in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}::operator()() const (/usr/bin/ceph-osd+0x2ca73306)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10662, detail=PGAdvanceMap(pg=9.1 from=46 to=48)): complete
ceph#10 0x5653d3e888b2 in std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}>::_M_invoke(std::_Any_data const&) (/usr/bin/ceph-osd+0x2ca738b2)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10680, detail=PGAdvanceMap(pg=12.4 from=46 to=48)): sending pg temp
ceph#11 0x5653d39775cf in seastar::posix_thread::start_routine(void*) (/usr/bin/ceph-osd+0x2c5625cf)
DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): complete
ceph#12 0x7f106029f801 in start_thread (/lib64/libc.so.6+0x9f801)
ceph#13 0x7f106023f44f in __GI___clone3 (/lib64/libc.so.6+0x3f44f)
DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): exit
DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - pg_advance_map(id=33559863, detail=PGAdvanceMap(pg=21.e from=46 to=48)): complete
0x6150002c9a5c is located 92 bytes inside of 472-byte region [0x6150002c9a00,0x6150002c9bd8)
DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 18.9
freed by thread T1 here:
#0 0x7f10628b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x5653c8794bff in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x2137fbff)
previously allocated by thread T1 here:
#0 0x7f10628b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
Thread T1 created by T0 here:
#0 0x7f10628587d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map : Not dirtying info: last_persisted is 31 while current is 48
#1 0x5653d3b89a69 in seastar::posix_thread::posix_thread(seastar::posix_thread::attr, std::function<void ()>) (/usr/bin/ceph-osd+0x2c774a69)
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - log is not dirty
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1ed27071) in seastar::shared_mutex::unlock()
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10821, detail=PGAdvanceMap(pg=48.3 from=46 to=48)): exit
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10733, detail=PGAdvanceMap(pg=22.11 from=46 to=48)): exit
```
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
Matan-B
added a commit
to Matan-B/ceph
that referenced
this pull request
Nov 15, 2023
```
DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - ShardServices::dispatch_context_transaction: empty transaction
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - do_peering_event ignoring epoch_sent: 37 epoch_requested: 37 RenewLease -- pg has reset
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - peering_event(id=33559971, detail=PeeringEvent(from=0 pgid=42.9 sent=37 requested=37 evt=epoch_sent: 37 epoch_requested: 37 RenewLease)): submitting ctx
DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - ShardServices::dispatch_context_transaction: empty transaction
#0 0x5653c613c071 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1ed27071)
DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - snaptrim_event(id=10610, detail=SnapTrimEvent(pgid=16.1a snapid=a needs_pause=0)): interrupted crimson::common::actingset_changed (acting set changed)
DEBUG 2023-11-14 15:23:50,927 [shard 0] osd - pg_advance_map(id=10742, detail=PGAdvanceMap(pg=24.2 from=46 to=48)): complete
#1 0x5653c8670acf in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x2125bacf)
#2 0x5653c8670e22 in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ (/usr/bin/ceph-osd+0x2125be22)
ceph#3 0x5653c5d0b4b2 in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) (/usr/bin/ceph-osd+0x1e8f64b2)
ceph#4 0x5653c5d3069f in void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) (/usr/bin/ceph-osd+0x1e91b69f)
ceph#5 0x5653c5d4e09d in seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x1e93909d)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map
ceph#6 0x5653d3bc3e2b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c7aee2b)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map : Not dirtying info: last_persisted is 28 while current is 48
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - log is not dirty
ceph#7 0x5653d3c61c42 in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c84cc42)
ceph#8 0x5653d3e84afa in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2ca6fafa)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10706, detail=PGAdvanceMap(pg=17.17 from=46 to=48)): exit
ceph#9 0x5653d3e88306 in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}::operator()() const (/usr/bin/ceph-osd+0x2ca73306)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10662, detail=PGAdvanceMap(pg=9.1 from=46 to=48)): complete
ceph#10 0x5653d3e888b2 in std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}>::_M_invoke(std::_Any_data const&) (/usr/bin/ceph-osd+0x2ca738b2)
DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10680, detail=PGAdvanceMap(pg=12.4 from=46 to=48)): sending pg temp
ceph#11 0x5653d39775cf in seastar::posix_thread::start_routine(void*) (/usr/bin/ceph-osd+0x2c5625cf)
DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): complete
ceph#12 0x7f106029f801 in start_thread (/lib64/libc.so.6+0x9f801)
ceph#13 0x7f106023f44f in __GI___clone3 (/lib64/libc.so.6+0x3f44f)
DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): exit
DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - pg_advance_map(id=33559863, detail=PGAdvanceMap(pg=21.e from=46 to=48)): complete
0x6150002c9a5c is located 92 bytes inside of 472-byte region [0x6150002c9a00,0x6150002c9bd8)
DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 18.9
freed by thread T1 here:
#0 0x7f10628b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
#1 0x5653c8794bff in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x2137fbff)
previously allocated by thread T1 here:
#0 0x7f10628b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
Thread T1 created by T0 here:
#0 0x7f10628587d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map : Not dirtying info: last_persisted is 31 while current is 48
#1 0x5653d3b89a69 in seastar::posix_thread::posix_thread(seastar::posix_thread::attr, std::function<void ()>) (/usr/bin/ceph-osd+0x2c774a69)
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - log is not dirty
SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1ed27071) in seastar::shared_mutex::unlock()
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10821, detail=PGAdvanceMap(pg=48.3 from=46 to=48)): exit
DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10733, detail=PGAdvanceMap(pg=22.11 from=46 to=48)): exit
```
Signed-off-by: Matan Breizman <mbreizma@redhat.com>
14 tasks
tchaikov
added a commit
that referenced
this pull request
Mar 25, 2024
before this change, we allocate memory chunks with specified
size using `new []`, but we never free them. when testing with
LeakSanitizer enabled, it rightly points identifies the leakage:
```
Direct leak of 8754 byte(s) in 184 object(s) allocated from:
#0 0x55c0b2470f0d in operator new[](unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_memory+0x196f0d) (BuildId: d3267dd8819427b804c4729e0467dbe7601fb321)
#1 0x55c0b247456c in MemoryIsZeroSmallTest_MemoryIsZeroTestSmall_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/common/test_memory.cc:33:18
#2 0x55c0b2598ee6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#3 0x55c0b2553b92 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#4 0x55c0b25049dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#5 0x55c0b2506a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#6 0x55c0b250804b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#7 0x55c0b25254d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#8 0x55c0b25a16f6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#9 0x55c0b255a502 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#10 0x55c0b2524862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#11 0x55c0b24ab4c0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#12 0x55c0b24ab451 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10
#13 0x7f45e065ad8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
in this change, we free the allocate memory.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 25, 2024
before this change, we increment the refcount when constructing `cct` instrusive_ptr, but nobody owns this smart pointer. also, `CephContext` 's constructor set its refcount to 1. so, when the test finishes, the refcount is 1, and this leads to a leakage of the `CephContext` instance, this not only annoys ASan, and defeats the purpose of 14d878c. ``` Indirect leak of 10880000 byte(s) in 1 object(s) allocated from: #0 0x5564d173537d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_ipaddr+0x19b37d) (BuildId: 45c0c7f28b253c04fcb7bb1a43aed52a5526d734) #1 0x7fe7f2ccd189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27 #2 0x7fe7f2ccc563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32 #3 0x7fe7f2ccc563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39 #4 0x7fe7f2ccc2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18 #5 0x7fe7f2cc6192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9 #6 0x7fe7f2cb91e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5 #7 0x7fe7f1f8f96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16 #8 0x7fe7f1f8e93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5 #9 0x5564d1752eb9 in pick_address_find_ip_in_subnet_list_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/test_ipaddr.cc:706:47 #10 0x5564d18694d6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 #11 0x5564d1820fc2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 #12 0x5564d17d19dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5 #13 0x5564d17d3a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11 #14 0x5564d17d504b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28 #15 0x5564d17f24d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44 #16 0x5564d1871d06 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10 #17 0x5564d1827932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14 #18 0x5564d17f1862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10 #19 0x5564d1775d80 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46 #20 0x5564d1775d11 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10 ``` so, in this change, we do not increase the refcount when creating cct. the same applies to `test/common/test_fault_injector.cc`. Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 25, 2024
before this change, in test_util.cc, we increment the refcount of
when constructing it. but at that moment, nobody really owns it.
also, `CephContext` 's refcount is set to 1 in its constructor.
so, we should not do this. otherwise, the created `CephContext`
is leaked as LeakSanitizer rightly points out:
```
Indirect leak of 10880000 byte(s) in 1 object(s) allocated from:
#0 0x5632320d27ed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_util+0x1917ed) (BuildId: ff1df1455bd07b651ad580584a17ea204afeb36e)
#1 0x7ff9d535b189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0x7ff9d535a563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32
#3 0x7ff9d535a563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39
#4 0x7ff9d535a2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18
#5 0x7ff9d5354192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9
#6 0x7ff9d53471e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5
#7 0x7ff9d461d96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16
#8 0x7ff9d461c93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5
#9 0x5632320d52e0 in util_collect_sys_info_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/common/test_util.cc:34:27
#10 0x563232205c16 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#11 0x5632321c2742 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#12 0x5632321736dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
```
in this change, instead of using a raw pointer, let's
use `boost::intrusive_ptr<CephContext>` to manage the lifecyle
of `CephContext`, this also address the leakage reported by
LeakSanitizer.
the same applies to common/test_context.cc
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 26, 2024
…sive_ptr<CephContext>
before this change, we increment the refcount when constructing
`cct` instrusive_ptr, but nobody owns this smart pointer. also,
`CephContext` 's constructor set its refcount to 1. so, when the
test finishes, the refcount is 1, and this leads to a leakage of
the `CephContext` instance. and LeakSanitizer points this out:
```
Indirect leak of 10880000 byte(s) in 1 object(s) allocated from:
#0 0xaaaac359c7c8 in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/bin/unittest_rgw_iam_policy+0x211c7c8) (BuildId: 060fadb10da261b52fd5757c7b1e9812d34542f1)
#1 0xffff96f764e4 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0xffff96f757cc in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/aarch64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32
#3 0xffff96f757cc in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:2396:39
#4 0xffff96f75500 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:2494:18
#5 0xffff96f6ec4c in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/build/boost/include/boost/circular_buffer/base.hpp:1039:9
#6 0xffff96f63528 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/log/Log.cc:53:5
#7 0xffff96045300 in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/common/ceph_context.cc:729:16
#8 0xffff960446ec in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/common/ceph_context.cc:697:5
#9 0xaaaac3629238 in IPPolicyTest::IPPolicyTest() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/rgw/test_rgw_iam_policy.cc:864:15
#10 0xaaaac3628da0 in IPPolicyTest_MaskedIPOperations_Test::IPPolicyTest_MaskedIPOperations_Test() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/test/rgw/test_rgw_iam_policy.cc:869:1
#11 0xaaaac3628d3c in testing::internal::TestFactoryImpl<IPPolicyTest_MaskedIPOperations_Test>::CreateTest() /home/jenkins-build/build/workspace/ceph-pull-requests-arm64/src/googletest/googletest/include/gtest/internal/gtest-internal.h:472:44
```
so, in this change, we do not increase the refcount when creating cct.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 26, 2024
before this change, we create a new cct instance with `new`, but
we never free this instance after done with it. and LeakSanitizer
points this out:
```
Indirect leak of 10880000 byte(s) in 1 object(s) allocated from:
#0 0x561afe148fed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_config_map+0x1c2fed) (BuildId: 3ce9eeed38cee335628fa74fdd08cd215b15019e)
#1 0x7f37dc9ac189 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0x7f37dc9ab563 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32
#3 0x7f37dc9ab563 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39
#4 0x7f37dc9ab2c0 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18
#5 0x7f37dc9a5192 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circular_buffer/base.hpp:1039:9
#6 0x7f37dc9981e4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5
#7 0x7f37dbc6e96d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16
#8 0x7f37dbc6d93b in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5
#9 0x561afe14e983 in ConfigMap_add_option_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_config_map.cc:58:18
#10 0x561afe2689b6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#11 0x561afe221262 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#12 0x561afe1d1f7c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#13 0x561afe1d3fb2 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#14 0x561afe1d55eb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#15 0x561afe1f2a78 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#16 0x561afe2711e6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#17 0x561afe227bd2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#18 0x561afe1f1e02 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#19 0x561afe176ec0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#20 0x561afe176e51 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10
#21 0x7f37d9397d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
so in this change, we manage the `CephContext` pointer with a smart
pointer. because the size of CephContext could be large, we don't create
it on stack.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 26, 2024
before this change, we create a new CrushWrapper instance with `new`, but
we never free this instance after done with it. and LeakSanitizer
points this out:
```
Direct leak of 544 byte(s) in 1 object(s) allocated from:
#0 0x561afe148fed in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_config_map+0x1c2fed) (BuildId: 3ce9eeed38cee335628fa74fdd08cd215b15019e)
#1 0x561afe151cbd in ConfigMap_result_sections_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/mon/test_config_map.cc:93:16
#2 0x561afe2689b6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#3 0x561afe221262 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#4 0x561afe1d1f7c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#5 0x561afe1d3fb2 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#6 0x561afe1d55eb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#7 0x561afe1f2a78 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#8 0x561afe2711e6 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#9 0x561afe227bd2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#10 0x561afe1f1e02 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#11 0x561afe176ec0 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#12 0x561afe176e51 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10
#13 0x7f37d9397d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
so in this change, we manage the `CrushWrapper` pointer with a smart
pointer. because the size of `CrushWrapper` is relatively large, we
don't create it on stack.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 29, 2024
before this change, we increment the refcount when constructing
`cct` instrusive_ptr, but nobody owns this smart pointer. also,
`CephContext` 's constructor set its refcount to 1. so, when the
test finishes, the refcount is 1, and this leads to a leakage of
the `CephContext` instance. and LeakSanitizer points this out:
```
Indirect leak of 10880000 byte(s) in 1 object(s) allocated from:
#0 0x558d341d837d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_ipaddr+0x19b37d) (BuildId: 1b7e7e5abfc2b58ce2334712e4c00b2441c25870)
#1 0x7fd74c957559 in __gnu_cxx::new_allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long, void const*) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/ext/new_allocator.h:127:27
#2 0x7fd74c956933 in std::allocator<ceph::logging::ConcreteEntry>::allocate(unsigned long) /usr/bin/../lib/gcc/x86_64-linux-gnu/11/../../../../include/c++/11/bits/allocator.h:185:32
#3 0x7fd74c956933 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::allocate(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2396:39
#4 0x7fd74c956690 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::initialize_buffer(unsigned long) /opt/ceph/include/boost/circular_buffer/base.hpp:2494:18
#5 0x7fd74c950562 in boost::circular_buffer<ceph::logging::ConcreteEntry, std::allocator<ceph::logging::ConcreteEntry> >::circular_buffer(unsigned long, std::allocator<ceph::logging::ConcreteEntry> const&) /opt/ceph/include/boost/circ
ular_buffer/base.hpp:1039:9
#6 0x7fd74c9435b4 in ceph::logging::Log::Log(ceph::logging::SubsystemMap const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/log/Log.cc:53:5
#7 0x7fd74bc1891d in ceph::common::CephContext::CephContext(unsigned int, ceph::common::CephContext::create_options const&) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:729:16
#8 0x7fd74bc178eb in ceph::common::CephContext::CephContext(unsigned int, code_environment_t, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/ceph_context.cc:697:5
#9 0x558d341f97e9 in pick_address_filtering_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/test_ipaddr.cc:774:47
#10 0x558d3430c4f6 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#11 0x558d342c3fc2 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#12 0x558d342749dc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#13 0x558d34276a12 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#14 0x558d3427804b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#15 0x558d342954d8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#16 0x558d34314d26 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#17 0x558d342ca932 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#18 0x558d34294862 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#19 0x558d34218d80 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#20 0x558d34218d11 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googlemock/src/gmock_main.cc:70:10
#21 0x7fd749331d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
so, in this change, we do not increase the refcount when creating cct.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 30, 2024
in BlueFS.test_shared_alloc, we keep the return value of
`fs.get_perf_counters()`, and deference it after umounting the fs,
but the `PerfCounters*` pointer returned from `fs.get_perf_counters()`
is destroyed in `BlueFS::_shutdown_logger()` which is in turn called
by `BlueFS::umount()`. so ASan points this out:
```
==1662613==ERROR: AddressSanitizer: heap-use-after-free on address 0x6110000b2d80 at pc 0x7f0eefc30644 bp 0x7ffcdbab6430 sp 0x7ffcdbab6428
READ of size 8 at 0x6110000b2d80 thread T0
#0 0x7f0eefc30643 in ceph::common::PerfCounters::get(int) const /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:246:8
#1 0x557595ddfc15 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1182:3
#2 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#3 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#4 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#5 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#6 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#7 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#8 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#9 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#10 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#11 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#12 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#13 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
#14 0x7f0eed083e3f in __libc_start_main csu/../csu/libc-start.c:392:3
#15 0x557595cd46a4 in _start (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x2856a4) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
0x6110000b2d80 is located 0 bytes inside of 208-byte region [0x6110000b2d80,0x6110000b2e50)
freed by thread T0 here:
#0 0x557595d92b1d in operator delete(void*) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x343b1d) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
#1 0x557595f31c43 in BlueFS::_shutdown_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:462:3
#2 0x557595f54ab5 in BlueFS::umount(bool) /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:1076:3
#3 0x557595ddfbd7 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1180:6
#4 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#5 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#6 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#7 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#8 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#9 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#10 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#11 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#12 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#13 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#14 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#15 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
previously allocated by thread T0 here:
#0 0x557595d922bd in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs+0x3432bd) (BuildId: 5439261504ca3d7549fe9bcda1d17ef6d4d9b644)
#1 0x7f0eefc33180 in ceph::common::PerfCountersBuilder::PerfCountersBuilder(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, int, int) /home/jenkins-build/build/workspace/ceph-pull-requests/src/common/perf_counters.cc:537:21
#2 0x557595f30ac9 in BlueFS::_init_logger() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:221:23
#3 0x557595f42bc6 in BlueFS::mount() /home/jenkins-build/build/workspace/ceph-pull-requests/src/os/bluestore/BlueFS.cc:977:3
#4 0x557595ddd339 in BlueFS_test_shared_alloc_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1139:3
#5 0x557595eeef66 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#6 0x557595ea8b22 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#7 0x557595e5974c in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#8 0x557595e5b782 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#9 0x557595e5cdbb in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#10 0x557595e7a248 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#11 0x557595ef7816 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#12 0x557595eaf5c2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#13 0x557595e795d2 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#14 0x557595e05370 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#15 0x557595dfc1f5 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs.cc:1603:10
#16 0x7f0eed083d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
in this change, instead of keeping `logger` across the `umount()` and
`mount()` calls, we get another instance of `logger`, query it for
the perf counter that we are interested, and compare the value
to see if it is unchanged.
this should address the ASan warning above.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
tchaikov
added a commit
that referenced
this pull request
Mar 31, 2024
before this change, we allocate an instance of `RocksDBStore` with
`new`, but we never free it. and LeanSanitizer points this out:
```
Direct leak of 952 byte(s) in 1 object(s) allocated from:
#0 0x55f31440bc2d in operator new(unsigned long) (/home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_rocksdb_option+0xaebc2d) (BuildId: 81b849dbc41cbc6b05d5e603d9ba8a002dab2d24)
#1 0x55f3144132fd in RocksDBOption_simple_Test::TestBody() /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/TestRocksdbOptionParse.cc:17:22
#2 0x55f3144ecf26 in void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#3 0x55f3144a4312 in void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#4 0x55f314453ccc in testing::Test::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2680:5
#5 0x55f314455d02 in testing::TestInfo::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2858:11
#6 0x55f31445733b in testing::TestSuite::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:3012:28
#7 0x55f3144747c8 in testing::internal::UnitTestImpl::RunAllTests() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5723:44
#8 0x55f3144f5576 in bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2605:10
#9 0x55f3144ab1a2 in bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*) /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:2641:14
#10 0x55f314473b52 in testing::UnitTest::Run() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/src/gtest.cc:5306:10
#11 0x55f31440f690 in RUN_ALL_TESTS() /home/jenkins-build/build/workspace/ceph-pull-requests/src/googletest/googletest/include/gtest/gtest.h:2486:46
#12 0x55f31440e4c3 in main /home/jenkins-build/build/workspace/ceph-pull-requests/src/test/unit.cc:45:10
#13 0x7f0d32551d8f in __libc_start_call_main csu/../sysdeps/nptl/libc_start_call_main.h:58:16
```
in this change, we manage the life cycle of `RocksDBStore` using
a smart pointer. this should address the leak.
Signed-off-by: Kefu Chai <tchaikov@gmail.com>
14 tasks
14 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use upstart event instead of "start"