osd: Adding const to methods in OSDMap by adamcrume · Pull Request #11 · ceph/ceph

adamcrume · 2012-05-31T19:54:19Z

Signed-off-by: Adam Crume adamcrume@gmail.com

Signed-off-by: Adam Crume <adamcrume@gmail.com>

liewegas · 2012-05-31T21:18:30Z

weird, that didn't actually merge anything.

i just cherry-picked manually instead!

Before the mon, and lockdep, in particular. #0 __pthread_mutex_lock (mutex=0x30) at pthread_mutex_lock.c:50 #1 0x0000000000816092 in ceph::log::Log::submit_entry (this=0x0, e=0x2f4a270) at log/Log.cc:138 #2 0x00000000007ee0f8 in handle_fatal_signal (signum=11) at global/signal_handler.cc:100 #3 <signal handler called> #4 0x00000000008e1300 in lockdep_will_lock (name=0x959aa7 "SignalHandler::lock", id=17) at common/lockdep.cc:163 #5 0x00000000008867fc in Mutex::_will_lock (this=0x2f20428) at ./common/Mutex.h:56 #6 0x0000000000886605 in Mutex::Lock (this=0x2f20428, no_lockdep=false) at common/Mutex.cc:81 #7 0x00000000007eeb95 in SignalHandler::entry (this=0x2f20300) at global/signal_handler.cc:198 #8 0x00000000008b0bd1 in Thread::_entry_func (arg=0x2f20300) at common/Thread.cc:43 #9 0x00007f36fefd6b50 in start_thread (arg=<optimized out>) at pthread_create.c:304 #10 0x00007f36fd80b6dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:112 #11 0x0000000000000000 in ?? () #0 0x00007f36fefd7e75 in pthread_join (threadid=139874129766144, thread_return=0x0) at pthread_join.c:89 #1 0x00000000008b11ec in Thread::join (this=0x2f20300, prval=0x0) at common/Thread.cc:130 #2 0x00000000007eeae7 in SignalHandler::shutdown (this=0x2f20300) at global/signal_handler.cc:186 #3 0x00000000007ee9cf in SignalHandler::~SignalHandler (this=0x2f20300, __in_chrg=<optimized out>) at global/signal_handler.cc:175 #4 0x00000000007eea58 in SignalHandler::~SignalHandler (this=0x2f20300, __in_chrg=<optimized out>) at global/signal_handler.cc:176 #5 0x00000000007ee643 in shutdown_async_signal_handler () at global/signal_handler.cc:324 #6 0x00000000006de9d2 in main (argc=7, argv=0x7fffbfb8a1e8) at ceph_mon.cc:439 Signed-off-by: Sage Weil <sage@inktank.com>

OSD: extent compare and write same operations

…ocks. Summary: SizeBeingCompacted was called without any lock protection. This causes crashes, especially when running db_bench with value_size=128K. The fix is to compute SizeUnderCompaction while holding the mutex and passing in these values into the call to Finalize. (gdb) where ceph#4 leveldb::VersionSet::SizeBeingCompacted (this=this@entry=0x7f0b490931c0, level=level@entry=4) at db/version_set.cc:1827 ceph#5 0x000000000043a3c8 in leveldb::VersionSet::Finalize (this=this@entry=0x7f0b490931c0, v=v@entry=0x7f0b3b86b480) at db/version_set.cc:1420 ceph#6 0x00000000004418d1 in leveldb::VersionSet::LogAndApply (this=0x7f0b490931c0, edit=0x7f0b3dc8c200, mu=0x7f0b490835b0, new_descriptor_log=<optimized out>) at db/version_set.cc:1016 ceph#7 0x00000000004222b2 in leveldb::DBImpl::InstallCompactionResults (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1473 ceph#8 0x0000000000426027 in leveldb::DBImpl::DoCompactionWork (this=this@entry=0x7f0b49083400, compact=compact@entry=0x7f0b2b8330f0) at db/db_impl.cc:1757 ceph#9 0x0000000000426690 in leveldb::DBImpl::BackgroundCompaction (this=this@entry=0x7f0b49083400, madeProgress=madeProgress@entry=0x7f0b41bf2d1e, deletion_state=...) at db/db_impl.cc:1268 ceph#10 0x0000000000428f42 in leveldb::DBImpl::BackgroundCall (this=0x7f0b49083400) at db/db_impl.cc:1170 ceph#11 0x000000000045348e in BGThread (this=0x7f0b49023100) at util/env_posix.cc:941 ceph#12 leveldb::(anonymous namespace)::PosixEnv::BGThreadWrapper (arg=0x7f0b49023100) at util/env_posix.cc:874 ceph#13 0x00007f0b4a7cf10d in start_thread (arg=0x7f0b41bf3700) at pthread_create.c:301 ceph#14 0x00007f0b49b4b11d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:115 Test Plan: make check I am running db_bench with a value size of 128K to see if the segfault is fixed. Reviewers: MarkCallaghan, sheki, emayanke Reviewed By: sheki CC: leveldb Differential Revision: https://reviews.facebook.net/D9279

URL: facebook/rocksdb#11

filestore: separete ondisk finisher with apply finisher Reviewed-by: Haomai Wang <haomai@xsky.com>

…sting-on-bucket-removal rgw: user quota may not adjust on bucket removal

os/bluestore: Adds lacking methods in bluestore_compression_header_t …

…er instance the caller needs to check the nullity of the parameter before calling PK11_FreeSymKey or PK11_FreeSlot, otherwise if CryptoAESKeyHandler::init failed, we will hit a segfault as follows: #0 0x00007f76844f5a95 in PK11_FreeSymKey () from /lib64/libnss3.so ceph#1 0x00007f76586b6e49 in CryptoAESKeyHandler::~CryptoAESKeyHandler() () from /lib64/librados.so.2 ceph#2 0x00007f76586b5eea in CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&) () from /lib64/librados.so.2 ceph#3 0x00007f76586b4b9c in CryptoKey::_set_secret(int, ceph::buffer::ptr const&) () from /lib64/librados.so.2 ceph#4 0x00007f76586b4e95 in CryptoKey::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#5 0x00007f76586b7ee6 in KeyRing::set_modifier(char const*, char const*, EntityName&, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&) () from /lib64/librados.so.2 ceph#6 0x00007f76586b8882 in KeyRing::decode_plaintext(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#7 0x00007f76586b9803 in KeyRing::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 ceph#8 0x00007f76586b9a1f in KeyRing::load(CephContext*, std::string const&) () from /lib64/librados.so.2 ceph#9 0x00007f76586ba04b in KeyRing::from_ceph_context(CephContext*) () from /lib64/librados.so.2 ceph#10 0x00007f765852d0cd in MonClient::init() () from /lib64/librados.so.2 ceph#11 0x00007f76583c15f5 in librados::RadosClient::connect() () from /lib64/librados.so.2 ceph#12 0x00007f765838cb1c in rados_connect () from /lib64/librados.so.2 ... Signed-off-by: runsisi <runsisi@zte.com.cn>

…er instance the caller needs to check the nullity of the parameter before calling PK11_FreeSymKey or PK11_FreeSlot, otherwise if CryptoAESKeyHandler::init failed, we will hit a segfault as follows: #0 0x00007f76844f5a95 in PK11_FreeSymKey () from /lib64/libnss3.so #1 0x00007f76586b6e49 in CryptoAESKeyHandler::~CryptoAESKeyHandler() () from /lib64/librados.so.2 #2 0x00007f76586b5eea in CryptoAES::get_key_handler(ceph::buffer::ptr const&, std::string&) () from /lib64/librados.so.2 #3 0x00007f76586b4b9c in CryptoKey::_set_secret(int, ceph::buffer::ptr const&) () from /lib64/librados.so.2 #4 0x00007f76586b4e95 in CryptoKey::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #5 0x00007f76586b7ee6 in KeyRing::set_modifier(char const*, char const*, EntityName&, std::map<std::string, ceph::buffer::list, std::less<std::string>, std::allocator<std::pair<std::string const, ceph::buffer::list> > >&) () from /lib64/librados.so.2 #6 0x00007f76586b8882 in KeyRing::decode_plaintext(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #7 0x00007f76586b9803 in KeyRing::decode(ceph::buffer::list::iterator&) () from /lib64/librados.so.2 #8 0x00007f76586b9a1f in KeyRing::load(CephContext*, std::string const&) () from /lib64/librados.so.2 #9 0x00007f76586ba04b in KeyRing::from_ceph_context(CephContext*) () from /lib64/librados.so.2 #10 0x00007f765852d0cd in MonClient::init() () from /lib64/librados.so.2 #11 0x00007f76583c15f5 in librados::RadosClient::connect() () from /lib64/librados.so.2 #12 0x00007f765838cb1c in rados_connect () from /lib64/librados.so.2 ... Signed-off-by: runsisi <runsisi@zte.com.cn>

increase verbosity for OSDs for ceph-deploy tests

Adding percentile

…letion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client ceph#2: sends a list bucket to OSD #1 5. RGW client ceph#2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client ceph#2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (ceph#2) 8. OSD ceph#2: deletes the object 9. RGW client ceph#2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step ceph#5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

…letion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client ceph#2: sends a list bucket to OSD #1 5. RGW client ceph#2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client ceph#2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (ceph#2) 8. OSD ceph#2: deletes the object 9. RGW client ceph#2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step ceph#5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Resolves: rhbz#1500904 Signed-off-by: J. Eric Ivancich <ivancich@redhat.com> (cherry picked from commit b33f529) Signed-off-by: Matt Benjamin <mbenjamin@redhat.com>

…letion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client #2: sends a list bucket to OSD #1 5. RGW client #2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client #2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (#2) 8. OSD #2: deletes the object 9. RGW client #2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step ceph#5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com> (cherry picked from commit b33f529)

…letion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client ceph#2: sends a list bucket to OSD #1 5. RGW client ceph#2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client ceph#2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (ceph#2) 8. OSD ceph#2: deletes the object 9. RGW client ceph#2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step ceph#5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

mgr/dashboard_v2: Create proper text editor config files to maintain consistent coding …

…letion We have a race condition: 1. RGW client #1: requests an object be deleted. 2. RGW client #1: sends a prepare op to bucket index OSD #1. 3. OSD #1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client #2: sends a list bucket to OSD #1 5. RGW client #2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client #2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (#1) 7. RGW client #1: sends a delete object to object OSD (#2) 8. OSD #2: deletes the object 9. RGW client #2: sends a complete op to bucket index OSD (#1) 10. OSD #1: completes the op 11. OSD #1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step #5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Signed-off-by: J. Eric Ivancich <ivancich@redhat.com> (cherry picked from commit b33f529) Conflicts: (backported substantial changes only; omitted cleanups) src/cls/rgw/cls_rgw.cc src/rgw/rgw_rados.cc

…letion We have a race condition: 1. RGW client ceph#1: requests an object be deleted. 2. RGW client ceph#1: sends a prepare op to bucket index OSD ceph#1. 3. OSD ceph#1: prepares the op, adding pending ops to the bucket dir entry 4. RGW client ceph#2: sends a list bucket to OSD ceph#1 5. RGW client ceph#2: sees that there are pending operations on bucket dir entry, and calls check_disk_state 6. RGW client ceph#2: check_disk_state sees that the object still exists, so it sends CEPH_RGW_UPDATE to bucket index OSD (ceph#1) 7. RGW client ceph#1: sends a delete object to object OSD (ceph#2) 8. OSD ceph#2: deletes the object 9. RGW client ceph#2: sends a complete op to bucket index OSD (ceph#1) 10. OSD ceph#1: completes the op 11. OSD ceph#1: receives the CEPH_RGW_UPDATE and updates the bucket index entry, thereby **RECREATING** it Solution implemented: At step ceph#5 the object's dir entry exists. If we get to beginning of step ceph#11 and the object's dir entry no longer exists, we know that the dir entry was just actively being modified, and ignore the CEPH_RGW_UPDATE operation, thereby NOT recreating it. Resolves: rhbz#1530784 Signed-off-by: J. Eric Ivancich <ivancich@redhat.com>

warn if prerequisite for deploying haproxy and keepalived not met

Recetly the `crimson::mon::Client` has started resending messages when session is being reopened which may happen during recovery from a network issue. A crash documented below has been observed in teuthology testing. It looks the fixes in `mon::Client` unveiled a problem in the messenger -- it assumes that a `Message` instance shall not be sent twice. This stays in conflict with the behaviour of `mon::Client` about e.g. `MMonCommand`. ``` INFO 2021-03-02 14:29:01,192 [shard 0] monc - handle_mon_map: renewed tickets DEBUG 2021-03-02 14:29:01,192 [shard 0] ms - [osd.2(client) v2:172.21.15.57:6804/34494@55832 >> mon.0 v2:172.21.15.57:3300/0] <== #2 === auth_reply(proto 2 0 (0) Success) v1 (18) INFO 2021-03-02 14:29:01,192 [shard 0] monc - handle_auth_reply mon v2:172.21.15.57:6804/34494 => v2:172.21.15.57:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 INFO 2021-03-02 14:29:01,192 [shard 0] monc - handle_auth_reply INFO 2021-03-02 14:29:01,192 [shard 0] monc - do_auth_single: mon v2:172.21.15.57:6804/34494 => v2:172.21.15.57:3300/0 returns auth_reply(proto 2 0 (0) Success) v1: 0 ERROR 2021-03-02 14:29:01,192 [shard 0] none - /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/17.0.0-1345-g1dc78fd5/rpm/el8/BUILD/ceph-17.0.0-1345-g1dc78fd5/src/crimson/net/ProtocolV2.cc:1828 : In function 'crimson::net::ProtocolV2::do_sweep_messages(const std::deque<boost::intrusive_ptr<Message> >&, size_t, bool, std::optional<utime_t>, bool)::<lambda(const MessageRef&)>', ceph_assert(%s) !msg->get_seq() && "message already has seq" Aborting on shard 0. Backtrace: 0x00000000013c2bbc 0x0000000001384d10 0x0000000001385012 0x00000000013850d2 /lib64/libpthread.so.0+0x0000000000012b1f /lib64/libc.so.6+0x00000000000377fe /lib64/libc.so.6+0x0000000000021c34 0x00000000005e2e98 0x00000000005e2ee0 0x0000000000dfb215 0x0000000000def167 0x0000000000df0854 0x0000000000df100e 0x0000000000df165c 0x0000000000de97e9 0x0000000000d8c3fd 0x0000000000d8c5bf 0x0000000000d85ba5 0x0000000001381237 0x00000000013815a2 0x00000000013ae735 0x000000000134b1d7 0x0000000000661de5 /lib64/libc.so.6+0x00000000000237b2 0x00000000006b256d daemon-helper: command crashed with signal 6 ``` Processing the backtrace with thes`seastar-addr2line` confirms the hypothesis. ``` [Backtrace ceph#8] ceph::__ceph_assert_fail(ceph::assert_data const&) at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/common/assert.cc:14 [Backtrace ceph#9] operator() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/ProtocolV2.cc:1828 (inlined by) ?? at /opt/rh/gcc-toolset-9/root/usr/include/c++/9/bits/stl_algo.h:3876 (inlined by) crimson::net::ProtocolV2::do_sweep_messages(std::deque<boost::intrusive_ptr<Message>, std::allocator<boost::intrusive_ptr<Message> > > const&, unsigned lo ng, bool, std::optional<utime_t>, bool) at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/ProtocolV2.cc:1848 [Backtrace ceph#10] operator() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:235 [Backtrace ceph#11] crimson::net::Protocol::do_write_dispatch_sweep() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/future.hh:2135 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/loop.hh:118 (inlined by) crimson::net::Protocol::do_write_dispatch_sweep() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:217 [Backtrace ceph#12] operator() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:312 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/future.hh:2135 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/future.hh:2166 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/gate.hh:126 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/gate.hh:144 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/common/gated.h:38 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/common/gated.h:23 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:311 (inlined by) crimson::net::Protocol::write_event() at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:298 [Backtrace ceph#13] crimson::net::Protocol::send(boost::intrusive_ptr<Message>) at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/Protocol.cc:97 [Backtrace ceph#14] crimson::net::SocketConnection::send(boost::intrusive_ptr<Message>) at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/net/SocketConnection.cc:75 [Backtrace ceph#15] crimson::mon::Client::send_message(boost::intrusive_ptr<Message>) at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/mon/MonClient.cc:1024 [Backtrace ceph#16] operator()<crimson::mon::Client::mon_command_t> at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/crimson/mon/MonClient.cc:1045 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/future.hh:2135 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/future.hh:2166 (inlined by) ?? at /usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/loop.hh:549 (inlined by) parallel_for_each_impl<std::vector<crimson::mon::Client::mon_command_t>&, crimson::mon::Client::on_session_opened()::<lambda()>::<lambda(auto:82&)> > at $ usr/src/debug/ceph-17.0.0-1345.g1dc78fd5.el8.x86_64/src/seastar/include/seastar/core/loop.hh:594 ``` In classical OSD the `Message` serialization code is free from the assertion: ```cpp ssize_t ProtocolV2::write_message(Message *m, bool more) { FUNCTRACE(cct); ceph_assert(connection->center->in_thread()); m->set_seq(++out_seq); // ... } ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Otherwise, if we assert, we'll hang here: Thread 1 (Thread 0x7f74eba79580 (LWP 1688617)): #0 0x00007f74eb2aa529 in futex_wait (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0x7ffd642b4b30) at pthread_cond_destroy.c:54 #3 0x0000563ff2e5a891 in LibRadosService_StatusFormat_Test::TestBody (this=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:78 #4 0x0000563ff2e9dc3a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x563ff2ea72e4 "the test body", method=<optimized out>, object=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2605 #5 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x563ff422a6d0, method=<optimized out>, location=location@entry=0x563ff2ea72e4 "the test body") at ./src/googletest/googletest/src/gtest.cc:2641 #6 0x0000563ff2e908c3 in testing::Test::Run (this=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2680 #7 0x0000563ff2e90a25 in testing::TestInfo::Run (this=0x563ff41a3b70) at ./src/googletest/googletest/src/gtest.cc:2858 #8 0x0000563ff2e90ec1 in testing::TestSuite::Run (this=0x563ff41b6230) at ./src/googletest/googletest/src/gtest.cc:3012 #9 0x0000563ff2e92bdc in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at ./src/googletest/googletest/src/gtest.cc:5723 #10 0x0000563ff2e9e14a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x563ff2ea8728 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x563ff41a2d10) at ./src/googletest/googletest/src/gtest.cc:2605 #11 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563ff41a2d10, method=<optimized out>, location=location@entry=0x563ff2ea8728 "auxiliary test code (environments or event listeners)") at ./src/googletest/googletest/src/gtest.cc:2641 #12 0x0000563ff2e90ae8 in testing::UnitTest::Run (this=0x563ff30c0660 <testing::UnitTest::GetInstance()::instance>) at ./src/googletest/googletest/src/gtest.cc:5306 Signed-off-by: Sage Weil <sage@newdream.net>

Otherwise, if we assert, we'll hang here: Thread 1 (Thread 0x7f74eba79580 (LWP 1688617)): #0 0x00007f74eb2aa529 in futex_wait (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/unix/sysv/linux/futex-internal.h:61 #1 futex_wait_simple (private=<optimized out>, expected=132, futex_word=0x7ffd642b4b54) at ../sysdeps/nptl/futex-internal.h:135 #2 __pthread_cond_destroy (cond=0x7ffd642b4b30) at pthread_cond_destroy.c:54 #3 0x0000563ff2e5a891 in LibRadosService_StatusFormat_Test::TestBody (this=<optimized out>) at /usr/include/c++/7/bits/unique_ptr.h:78 #4 0x0000563ff2e9dc3a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void> (location=0x563ff2ea72e4 "the test body", method=<optimized out>, object=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2605 #5 testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void> (object=object@entry=0x563ff422a6d0, method=<optimized out>, location=location@entry=0x563ff2ea72e4 "the test body") at ./src/googletest/googletest/src/gtest.cc:2641 #6 0x0000563ff2e908c3 in testing::Test::Run (this=0x563ff422a6d0) at ./src/googletest/googletest/src/gtest.cc:2680 #7 0x0000563ff2e90a25 in testing::TestInfo::Run (this=0x563ff41a3b70) at ./src/googletest/googletest/src/gtest.cc:2858 #8 0x0000563ff2e90ec1 in testing::TestSuite::Run (this=0x563ff41b6230) at ./src/googletest/googletest/src/gtest.cc:3012 #9 0x0000563ff2e92bdc in testing::internal::UnitTestImpl::RunAllTests (this=<optimized out>) at ./src/googletest/googletest/src/gtest.cc:5723 #10 0x0000563ff2e9e14a in testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (location=0x563ff2ea8728 "auxiliary test code (environments or event listeners)", method=<optimized out>, object=0x563ff41a2d10) at ./src/googletest/googletest/src/gtest.cc:2605 #11 testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool> (object=0x563ff41a2d10, method=<optimized out>, location=location@entry=0x563ff2ea8728 "auxiliary test code (environments or event listeners)") at ./src/googletest/googletest/src/gtest.cc:2641 #12 0x0000563ff2e90ae8 in testing::UnitTest::Run (this=0x563ff30c0660 <testing::UnitTest::GetInstance()::instance>) at ./src/googletest/googletest/src/gtest.cc:5306 Signed-off-by: Sage Weil <sage@newdream.net> (cherry picked from commit ee5a0c9)

otherwise i have following ASan error when compiling the tree with ASan enabled. ==1086666==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7fffe896c364 at pc 0x7ffff76253ae bp 0x7fffe896c330 sp 0x7fffe896bae0 #0 0x7ffff76253ad in __interceptor_sigaltstack ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:9996 #1 0x7ffff7687163 in __asan::PlatformUnpoisonStacks() ../../../../src/libsanitizer/asan/asan_posix.cpp:44 #2 0x7ffff768be6c in __asan_handle_no_return ../../../../src/libsanitizer/asan/asan_rtl.cpp:612 #3 0x555570b14515 in EntityName::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) ../src/common/entity_name.h:39 #4 0x555570b14626 in decode(EntityName&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) ../src/common/entity_name.h:88 #5 0x555571e5f579 in std::enable_if<(!denc_traits<EntityName, void>::supported)||(!denc_traits<EntityAuth, void>::supported), void>::type ceph::decode<EntityName, EntityAuth, std::less<EntityName>, std::allocator<std::pair<EntityName const, EntityAuth> >, denc_traits< EntityName, void>, denc_traits<EntityAuth, void> >(std::map<EntityName, EntityAuth, std::less<EntityName>, std::allocator<std::pair<EntityName const, EntityAuth> > >&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) ../src/include/encoding.h:1046 #6 0x555571e5a637 in KeyRing::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) ../src/auth/KeyRing.cc:210 #7 0x555571e5b0e4 in KeyRing::load(crimson::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ../src/auth/KeyRing.cc:232 #8 0x555571e5438a in KeyRing::from_ceph_context(crimson::common::CephContext*) ../src/auth/KeyRing.cc:48 #9 0x5555721163b8 in AuthRegistry::_refresh_config() ../src/auth/AuthRegistry.cc:163 #10 0x555571efa019 in AuthRegistry::refresh_config() ../src/auth/AuthRegistry.h:46 #11 0x555571eae4fc in crimson::mon::Client::start() ../src/crimson/mon/MonClient.cc:423 #12 0x55556e87d73b in operator() ../src/crimson/osd/main.cc:160 ceph#13 0x55556e896b10 in __invoke_impl<void, fetch_config()::<lambda()> > /usr/include/c++/11/bits/invoke.h:61 ceph#14 0x55556e8934eb in __invoke<fetch_config()::<lambda()> > /usr/include/c++/11/bits/invoke.h:96 ceph#15 0x55556e88f2a3 in __apply_impl<fetch_config()::<lambda()>, std::tuple<> > /usr/include/c++/11/tuple:1806 ceph#16 0x55556e88f313 in apply<fetch_config()::<lambda()>, std::tuple<> > /usr/include/c++/11/tuple:1817 ceph#17 0x55556e88f3b4 in apply<fetch_config()::<lambda()> > ../src/seastar/include/seastar/core/future.hh:2099 ceph#18 0x55556e88980c in operator() ../src/seastar/include/seastar/core/thread.hh:258 ceph#19 0x55556e8995d7 in call ../src/seastar/include/seastar/util/noncopyable_function.hh:124 ceph#20 0x555574f5c8fe in seastar::noncopyable_function<void ()>::operator()() const ../src/seastar/include/seastar/util/noncopyable_function.hh:209 ceph#21 0x5555754089ea in seastar::thread_context::main() ../src/seastar/src/core/thread.cc:299 0x7fffe896c364 is located 246628 bytes inside of 262144-byte region [0x7fffe8930000,0x7fffe8970000) allocated by thread T0 here: #0 0x7ffff76825df in __interceptor_aligned_alloc ../../../../src/libsanitizer/asan/asan_malloc_linux.cpp:192 SUMMARY: AddressSanitizer: stack-buffer-overflow ../../../../src/libsanitizer/sanitizer_common/sanitizer_common_interceptors.inc:9996 in __interceptor_sigaltstack the root cause is that when we decode a KeyRing as a binary blob, we first decode the struct_v and then decode as remainder into a std::map<EntityName, EntityAuth>. if the buffer being decoded is a actually a plaintext, there is good chance the number of items of the key would be a huge number, and the decoder of map<> just following the instruction and try to decode all of them until reaching the end of buffer. but we don't actually check the boundary of bufferlist when decoding it, and we move across the boundary of the bufferlist, we are accessing the forbidden bits.. to workaround this issue, in this change, we try to decode the KeyRing as plaintext first, and if it fails to decode, we try to decode as a binary blob. this change does not address the ASan issue, it just alleviates it. unless we have a magic number in front of the bufferlist denoting if the keyring blob is in plaintext or binary, it's difficult to fully address this issue. but we have lots of keyring persisted in existing Ceph deployment, it might be difficult to enfoce the new keyring format outlined above. Signed-off-by: Kefu Chai <kchai@redhat.com>

``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) #1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) #2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 ceph#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 ceph#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 ceph#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 ceph#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 ceph#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) #1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 #2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

The problem is: ``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) #1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) #2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 ceph#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 ceph#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 ceph#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 ceph#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 ceph#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) #1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 #2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

The problem is: ``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) #1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) #2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 #3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 #4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 #5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 #6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 #7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 #8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 #9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 #10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 #11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) #1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 #2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

The problem is: ``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) #1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) #2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 #3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 #4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 #5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 #6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 #7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 #8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 #9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 #10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 #11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 #12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 #13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 #14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 #15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 #16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 #17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 #18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 #19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 #20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 #21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 #22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) #1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 #2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

The problem is: ``` DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - calling method rbd.create, num_read=0, num_write=0 DEBUG 2022-03-07 13:50:40,027 [shard 0] objclass - <cls> ../src/cls/rbd/cls_rbd.cc:787: create object_prefix=parent_id size=2097152 order=0 features=1 DEBUG 2022-03-07 13:50:40,027 [shard 0] osd - handling op omap-get-vals-by-keys on object 1:144d5af5:::parent_id:head ================================================================= ==2109764==ERROR: AddressSanitizer: stack-buffer-overflow on address 0x7f6de5176e70 at pc 0x7f6dfd2a7157 bp 0x7f6de5176e30 sp 0x7f6de51765d8 WRITE of size 24 at 0x7f6de5176e70 thread T0 #0 0x7f6dfd2a7156 in __interceptor_sigaltstack.part.0 (/lib64/libasan.so.6+0x54156) ceph#1 0x7f6dfd30d5b3 in __asan::PlatformUnpoisonStacks() (/lib64/libasan.so.6+0xba5b3) ceph#2 0x7f6dfd31314c in __asan_handle_no_return (/lib64/libasan.so.6+0xc014c) Reactor stalled for 275 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd3383c1 0x7f6dfd339b18 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd339bd4 0x7f6dfd33b089 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#3 0x1881f22 in fmt::v6::internal::arg_map<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~arg_map() /usr/include/fmt/core.h:1170 ceph#4 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::~basic_format_context() /usr/include/fmt/core.h:1265 ceph#5 0x1881f22 in fmt::v6::format_handler<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >::~format_handler() /usr/include/fmt/format.h:3143 ceph#6 0x1881f22 in fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char>::iterator fmt::v6::vformat_to<fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >, char, fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >(fmt::v6::arg_formatter<fmt::v6::internal::output_range<seastar::internal::log_buf::inserter_iterator, char> >::range, fmt::v6::basic_string_view<char>, fmt::v6::basic_format_args<fmt::v6::basic_format_context<seastar::internal::log_buf::inserter_iterator, char> >, fmt::v6::internal::locale_ref) /usr/include/fmt/format.h:3206 ceph#7 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::vformat_to<fmt::v6::basic_string_view<char>, seastar::internal::log_buf::inserter_iterator, , 0>(seastar::internal::log_buf::inserter_iterator, fmt::v6::basic_string_view<char> const&, fmt::v6::basic_format_args<fmt::v6::basic_format_context<fmt::v6::type_identity<seastar::internal::log_buf::inserter_iterator>::type, fmt::v6::internal::char_t_impl<fmt::v6::basic_string_view<char>, void>::type> >) /usr/include/fmt/format.h:3395 ceph#8 0x188344a in seastar::internal::log_buf::inserter_iterator fmt::v6::format_to<seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> >, hobject_t const&, 0>(seastar::internal::log_buf::inserter_iterator, std::basic_string_view<char, std::char_traits<char> > const&, hobject_t const&) /usr/include/fmt/format.h:3418 ceph#9 0x188344a in seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)ceph#1}::operator()(seastar::internal::log_buf::inserter_iterator) const ../src/seastar/include/seastar/util/log.hh:227 ceph#10 0x188344a in seastar::logger::lambda_log_writer<seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&)::{lambda(seastar::internal::log_buf::inserter_iterator)ceph#1}>::operator()(seastar::internal::log_buf::inserter_iterator) ../src/seastar/include/seastar/util/log.hh:106 ceph#11 0xe8b439d in operator() ../src/seastar/src/util/log.cc:268 ceph#12 0xe8b58f2 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) ../src/seastar/src/util/log.cc:280 ceph#13 0x2521d5a in void seastar::logger::log<hobject_t const&>(seastar::log_level, seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:230 ceph#14 0x2a2ee12 in void seastar::logger::debug<hobject_t const&>(seastar::logger::format_info, hobject_t const&) ../src/seastar/include/seastar/util/log.hh:373 ceph#15 0x2a2ee12 in PGBackend::omap_get_vals_by_keys(ObjectState const&, OSDOp&, object_stat_sum_t&) const ../src/crimson/osd/pg_backend.cc:1220 ceph#16 0x2c76349 in operator()<PGBackend, ObjectState> ../src/crimson/osd/ops_executer.cc:577 ceph#17 0x2c76349 in do_const_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.cc:449 ceph#18 0x2e04ce9 in do_read_op<crimson::osd::OpsExecuter::execute_op(OSDOp&)::<lambda(auto:167&, const auto:168&)> > ../src/crimson/osd/ops_executer.h:216 ceph#19 0x2e04ce9 in crimson::osd::OpsExecuter::execute_op(OSDOp&) ../src/crimson/osd/ops_executer.cc:576 Reactor stalled for 762 ms on shard 0. Backtrace: 0x45d9d 0xda72bd3 0xd801f73 0xd81f6f9 0xd81fb9c 0xd81fe2c 0xd8200f7 0x12b2f 0x7f6dfd33ae85 0x7f6dfd33bb36 0x7f6dfd32e0b5 0x7f6dfd32ff3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0x54178 0xba5b3 0xc014c 0x1881f22 0x188344a 0xe8b439d 0xe8b58f2 0x2521d5a 0x2a2ee12 0x2c76349 0x2e04ce9 0x3c70c55 0x3cb8aa8 0x7f6de558de39 ceph#20 0x3c70c55 in execute_osd_op ../src/crimson/osd/objclass.cc:35 ceph#21 0x3cb8aa8 in cls_cxx_map_get_val(void*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ceph::buffer::v15_2_0::list*) ../src/crimson/osd/objclass.cc:372 ceph#22 0x7f6de558de39 (/home/rzarzynski/ceph1/build/lib/libcls_rbd.so.1.0.0+0x28e39) 0x7f6de5176e70 is located 249456 bytes inside of 262144-byte region [0x7f6de513a000,0x7f6de517a000) allocated by thread T0 here: #0 0x7f6dfd3084a7 in aligned_alloc (/lib64/libasan.so.6+0xb54a7) ceph#1 0xdd414fc in seastar::thread_context::make_stack(unsigned long) ../src/seastar/src/core/thread.cc:196 ceph#2 0x7fff3214bc4f ([stack]+0xa5c4f) ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Before the patch there was a possibility that `OSDConnectionPriv` gets destructed before a `PipelineHandle` instance that was using it. The reason is our remote-handling operations store `conn` directly while `handle` is defined in a parent class. Due to the language rules the former gets deinitialized earlier. ``` ==756032==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000039684 at pc 0x0000020bdfa2 bp 0x7ffd3abfa370 sp 0x7ffd3abfa360 READ of size 1 at 0x615000039684 thread T0 Reactor stalled for 261 ms on shard 0. Backtrace: 0x45d9d 0xe90f6d1 0xe6b8a1d 0xe6d1205 0xe6d16a8 0xe6d1938 0xe6d1c03 0x12cdf 0xccebf 0x7f6447161b1e 0x7f644714aee8 0x7f644714eed6 0x7f644714fb36 0x7f64471420b5 0x 7f6447143f3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0xbdc1a 0x20bdfa1 0x20c184e 0x352eb7f 0x352fa28 0x20b04a5 0x1be30e5 0xe694bc4 0xe6ebb8a 0xe843a11 0xe845a22 0xe29f497 0xe2a3ccd 0x1ab1841 0x3aca2 0x175698d #0 0x20bdfa1 in seastar::shared_mutex::unlock() ../src/seastar/include/seastar/core/shared_mutex.hh:122 #1 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::exit() ../src/crimson/common/operation.h:548 #2 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::exit() ../src/crimson/common/operation.h:533 ceph#3 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::cancel() ../src/crimson/common/operation.h:539 ceph#4 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:543 ceph#5 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:544 ceph#6 0x352eb7f in std::default_delete<crimson::PipelineExitBarrierI>::operator()(crimson::PipelineExitBarrierI*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85 ceph#7 0x352eb7f in std::unique_ptr<crimson::PipelineExitBarrierI, std::default_delete<crimson::PipelineExitBarrierI> >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361 ceph#8 0x352eb7f in crimson::PipelineHandle::~PipelineHandle() ../src/crimson/common/operation.h:457 ceph#9 0x352eb7f in crimson::osd::PhasedOperationT<crimson::osd::ClientRequest>::~PhasedOperationT() ../src/crimson/osd/osd_operation.h:152 ceph#10 0x352eb7f in crimson::osd::ClientRequest::~ClientRequest() ../src/crimson/osd/osd_operations/client_request.cc:64 ceph#11 ... ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Before the patch there was a possibility that `OSDConnectionPriv` gets destructed before a `PipelineHandle` instance that was using it. The reason is our remote-handling operations store `conn` directly while `handle` is defined in a parent class. Due to the language rules the former gets deinitialized earlier. ``` ==756032==ERROR: AddressSanitizer: heap-use-after-free on address 0x615000039684 at pc 0x0000020bdfa2 bp 0x7ffd3abfa370 sp 0x7ffd3abfa360 READ of size 1 at 0x615000039684 thread T0 Reactor stalled for 261 ms on shard 0. Backtrace: 0x45d9d 0xe90f6d1 0xe6b8a1d 0xe6d1205 0xe6d16a8 0xe6d1938 0xe6d1c03 0x12cdf 0xccebf 0x7f6447161b1e 0x7f644714aee8 0x7f644714eed6 0x7f644714fb36 0x7f64471420b5 0x 7f6447143f3a 0xd61d0 0x32412 0xbd8a7 0xbd134 0xbdc1a 0x20bdfa1 0x20c184e 0x352eb7f 0x352fa28 0x20b04a5 0x1be30e5 0xe694bc4 0xe6ebb8a 0xe843a11 0xe845a22 0xe29f497 0xe2a3ccd 0x1ab1841 0x3aca2 0x175698d #0 0x20bdfa1 in seastar::shared_mutex::unlock() ../src/seastar/include/seastar/core/shared_mutex.hh:122 #1 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::exit() ../src/crimson/common/operation.h:548 #2 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::exit() ../src/crimson/common/operation.h:533 #3 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::cancel() ../src/crimson/common/operation.h:539 #4 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:543 #5 0x20c184e in crimson::OrderedExclusivePhaseT<crimson::osd::ConnectionPipeline::GetPG>::ExitBarrier::~ExitBarrier() ../src/crimson/common/operation.h:544 #6 0x352eb7f in std::default_delete<crimson::PipelineExitBarrierI>::operator()(crimson::PipelineExitBarrierI*) const /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:85 #7 0x352eb7f in std::unique_ptr<crimson::PipelineExitBarrierI, std::default_delete<crimson::PipelineExitBarrierI> >::~unique_ptr() /opt/rh/gcc-toolset-11/root/usr/include/c++/11/bits/unique_ptr.h:361 #8 0x352eb7f in crimson::PipelineHandle::~PipelineHandle() ../src/crimson/common/operation.h:457 #9 0x352eb7f in crimson::osd::PhasedOperationT<crimson::osd::ClientRequest>::~PhasedOperationT() ../src/crimson/osd/osd_operation.h:152 #10 0x352eb7f in crimson::osd::ClientRequest::~ClientRequest() ../src/crimson/osd/osd_operations/client_request.cc:64 #11 ... ``` Signed-off-by: Radoslaw Zarzynski <rzarzyns@redhat.com>

Call get_new_bucket_name in long bucket name tests Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

c4d30d7 Ravindra Choudhari Mon, 27 Jun 2022 removing region name 4a13f58 Ravindra Choudhari Thu, 16 Jun 2022 Updating readme file (ceph#15) 18bc152 Ravindra Choudhari Tue, 14 Jun 2022 Adding attr test_of_iam to all user policy tests (ceph#13) 03f520a Ravindra Choudhari Tue, 14 Jun 2022 resolving review comments (ceph#12) 7cf2823 Ravindra Choudhari Mon, 13 Jun 2022 added IAM policy test section in README.rst (ceph#11) 563f3ea Ravindra Choudhari Fri, 10 Jun 2022 adding failing three tests back with attr @fails_on_rgw (ceph#10) 696dd2e Ravindra Choudhari Mon, 6 Jun 2022 changes as per review comments 3d63dfd Ravindra Choudhari Mon, 6 Jun 2022 Fixed review comments (ceph#8) 9492f69 Ravindra Choudhari Fri, 3 Jun 2022 Fixed review comments (ceph#7) 74095dc Ketan Arlulkar Wed, 1 Jun 2022 Fixed review comments (ceph#6) 942fb4f Ketan Arlulkar Wed, 1 Jun 2022 Added Tests for conflicting policies and IAM actions (ceph#4) ad5b5ae Ravindra Choudhari Tue, 31 May 2022 IAM policies s3 actions (ceph#5) 6515ec6 Ketan Arlulkar Fri, 27 May 2022 Corrected eq import 40a2841 Ravindra Choudhari Tue, 17 May 2022 resolving conflicts f53a5c1 Ravindra Choudhari Tue, 17 May 2022 added cleanup 747d563 Ketan Arlulkar Tue, 17 May 2022 Added cleanup/Delete Policy d1cc1d8 Ketan Arlulkar Mon, 16 May 2022 Fixed review comments 1ec43a2 Ravindra Choudhari Mon, 16 May 2022 delete user policy tests a01722e Ravindra Choudhari Mon, 16 May 2022 get user policy tests ff9d676 Ketan Arlulkar Fri, 13 May 2022 Removed TEST IDs d261400 Ketan Arlulkar Tue, 10 May 2022 Put User Policy & List User Policy Tests Signed-off-by: Ravindra Choudhari <ravindra.choudhari@seagate.com>

Beacuse the loop's returned future is ignored, we should cover the scenario where the pg is removed and the snap_trimq iteration didn't complete yet. Spotted in testing: ``` ================================================================= ==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0 READ of size 8 at 0x625000f326d8 thread T0 Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: #0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172) #1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708) #2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf) ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc) ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198) ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046) ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196) ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe) ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc) ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050) ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1) ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03) ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b) ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45) ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac) ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712) ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b) ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c) ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e) ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281) ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2) ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb) ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8) ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f) ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44) 0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8) freed by thread T0 here: #0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf) #1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b) previously allocated by thread T0 here: #0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367) Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82 #1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6) #2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c) ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87) SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) ``` Signed-off-by: Matan Breizman <mbreizma@redhat.com>

Beacuse the loop's returned future is ignored, we should cover the scenario where the pg is removed and the snap_trimq iteration didn't complete yet. Fixes: https://tracker.ceph.com/issues/61653 Spotted in testing: ``` ================================================================= ==81009==ERROR: AddressSanitizer: heap-use-after-free on address 0x625000f326d8 at pc 0x55c0a5fc6173 bp 0x7ffdd9397c00 sp 0x7ffdd9397bf0 READ of size 8 at 0x625000f326d8 thread T0 Reactor stalled for 36 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf07f8eec 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf07f8fad 0x29fcf07f9022 0x29fcf080923f 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: Reactor stalled for 94 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0x29fcf0804ef3 0x29fcf0805a5e 0x29fcf080878a 0x29fcf0809410 0x29fcee2a52d3 0x2c2d1aa9 0x29fcf0809684 0x29fcf07f8be9 0x29fcf07f8cb5 0x29fcf07ea165 0x29fcf07ebfea 0xd6280 0x32402 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: #0 0x55c0a5fc6172 in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) (/usr/bin/ceph-osd+0x1f6d5172) #1 0x55c0a664d708 in void fmt::v9::detail::format_value<char, crimson::osd::PG>(fmt::v9::detail::buffer<char>&, crimson::osd::PG const&, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1fd5c708) #2 0x55c0a664ddbf in fmt::v9::appender fmt::v9::basic_ostream_formatter<char>::format<crimson::osd::PG, fmt::v9::appender>(crimson::osd::PG const&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) const (/usr/bin/ceph-osd+0x1fd5cdbf) ceph#3 0x55c0a664ecdc in void fmt::v9::detail::value<fmt::v9::basic_format_context<fmt::v9::appender, char> >::format_custom_arg<crimson::osd::PG, fmt::v9::formatter<crimson::osd::PG, char, void> >(void*, fmt::v9::basic_format_parse_context<char, fmt::v9::detail::error_handler>&, fmt::v9::basic_format_context<fmt::v9::appender, char>&) (/usr/bin/ceph-osd+0x1fd5dcdc) ceph#4 0x55c0a4f20198 in fmt::v9::detail::default_arg_formatter<char>::operator()(fmt::v9::basic_format_arg<fmt::v9::basic_format_context<fmt::v9::appender, char> >::handle) (/usr/bin/ceph-osd+0x1e62f198) ceph#5 0x55c0a505b046 in char const* fmt::v9::detail::parse_replacement_field<char, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&>(char const*, char const*, fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref)::format_handler&) (/usr/bin/ceph-osd+0x1e76a046) ceph#6 0x55c0a505f196 in void fmt::v9::detail::vformat_to<char>(fmt::v9::detail::buffer<char>&, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<std::conditional<std::is_same<fmt::v9::type_identity<char>::type, char>::value, fmt::v9::appender, std::back_insert_iterator<fmt::v9::detail::buffer<fmt::v9::type_identity<char>::type> > >::type, fmt::v9::type_identity<char>::type> >, fmt::v9::detail::locale_ref) (/usr/bin/ceph-osd+0x1e76e196) ceph#7 0x55c0a505fbfe in seastar::internal::log_buf::inserter_iterator fmt::v9::vformat_to<seastar::internal::log_buf::inserter_iterator, 0>(seastar::internal::log_buf::inserter_iterator, fmt::v9::basic_string_view<char>, fmt::v9::basic_format_args<fmt::v9::basic_format_context<fmt::v9::appender, char> >) (/usr/bin/ceph-osd+0x1e76ebfe) ceph#8 0x55c0a65431dc in seastar::logger::lambda_log_writer<seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}>::operator()(seastar::internal::log_buf::inserter_iterator) (/usr/bin/ceph-osd+0x1fc521dc) ceph#9 0x55c0b3b46050 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&)::{lambda(seastar::internal::log_buf::inserter_iterator)#1}::operator()(seastar::internal::log_buf::inserter_iterator) const (/usr/bin/ceph-osd+0x2d255050) ceph#10 0x55c0b3c8f3d1 in seastar::logger::do_log(seastar::log_level, seastar::logger::log_writer&) (/usr/bin/ceph-osd+0x2d39e3d1) ceph#11 0x55c0a616bd03 in void seastar::logger::log<crimson::osd::PG&>(seastar::log_level, seastar::logger::format_info, crimson::osd::PG&) (/usr/bin/ceph-osd+0x1f87ad03) ceph#12 0x55c0a5f5b03b in _ZN7crimson9erroratorIJNS_19unthrowable_wrapperIRKSt10error_codeL_ZNS_2ecILi2EEEEEENS1_IS4_L_ZNS5_ILi11EEEEEEEE7_futureINS_23errorated_future_markerIN7seastar10bool_classINSB_18stop_iteration_tagEEEEEE24_safe_then_handle_errorsINSB_8futurizeINSB_6futureISE_EEEESK_ZNS_L8composerIZNS6_6handleIZZZNS_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKS6_E_JZNS7_6handleIZZZNSP_16on_active_actmapEvENKSQ_clEvENKSR_clEvEUlvE0_EEDaSU_EUlRKS7_E_EEEDaSU_DpOT0_EUlDpOT_E_EEDaOT0_OT1_.lto_priv.0 (/usr/bin/ceph-osd+0x1f66a03b) ceph#13 0x55c0a64e9c45 in _ZN7seastar20noncopyable_functionIFNS_6futureINS_10bool_classINS_18stop_iteration_tagEEEEEOS5_EE17direct_vtable_forIZNS5_24then_wrapped_maybe_eraseILb0ES5_ZN7crimson9erroratorIJNSB_19unthrowable_wrapperIRKSt10error_codeL_ZNSB_2ecILi2EEEEEENSD_ISG_L_ZNSH_ILi11EEEEEEEE7_futureINSB_23errorated_future_markerIS4_EEE12handle_errorIZNSB_L8composerIZNSI_6handleIZZZNSB_3osd2PG16on_active_actmapEvENKUlvE0_clEvENKUlvE_clEvEUlvE_EEDaOT_EUlRKSI_E_JZNSJ_6handleIZZZNST_16on_active_actmapEvENKSU_clEvENKSV_clEvEUlvE0_EEDaSY_EUlRKSJ_E_EEEDaSY_DpOT0_EUlDpOT_E_EEDaSY_EUlSY_E_EENS_8futurizeIT0_E4typeEOT1_EUlS6_E_E4callEPKS8_S6_.lto_priv.0 (/usr/bin/ceph-osd+0x1fbf8c45) ceph#14 0x55c0a65cc5ac in void seastar::futurize<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > >::satisfy_with_result_of<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}::operator()(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&) const::{lambda()#1}&&) (/usr/bin/ceph-osd+0x1fcdb5ac) ceph#15 0x55c0a65cd712 in seastar::continuation<seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>, seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >::then_wrapped_nrvo<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)> >(seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&&)::{lambda(seastar::internal::promise_base_with_type<seastar::bool_class<seastar::stop_iteration_tag> >&&, seastar::noncopyable_function<seastar::future<seastar::bool_class<seastar::stop_iteration_tag> > (seastar::future<seastar::bool_class<seastar::stop_iteration_tag> >&&)>&, seastar::future_state<seastar::bool_class<seastar::stop_iteration_tag> >&&)#1}, seastar::bool_class<seastar::stop_iteration_tag> >::run_and_dispose() (/usr/bin/ceph-osd+0x1fcdc712) ceph#16 0x55c0b2f1395b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c62295b) ceph#17 0x55c0b2fad51c in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c6bc51c) ceph#18 0x55c0b31cb55e in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2c8da55e) ceph#19 0x55c0b31cd281 in seastar::reactor::run() (/usr/bin/ceph-osd+0x2c8dc281) ceph#20 0x55c0b2c264f2 in seastar::app_template::run_deprecated(int, char**, std::function<void ()>&&) (/usr/bin/ceph-osd+0x2c3354f2) ceph#21 0x55c0b2c283fb in seastar::app_template::run(int, char**, std::function<seastar::future<int> ()>&&) (/usr/bin/ceph-osd+0x2c3373fb) ceph#22 0x55c0a54736c8 in main (/usr/bin/ceph-osd+0x1eb826c8) ceph#23 0x7fbd74a3feaf in __libc_start_call_main (/lib64/libc.so.6+0x3feaf) ceph#24 0x7fbd74a3ff5f in __libc_start_main_impl (/lib64/libc.so.6+0x3ff5f) ceph#25 0x55c0a4f1ca44 in _start (/usr/bin/ceph-osd+0x1e62ba44) 0x625000f326d8 is located 1496 bytes inside of 9144-byte region [0x625000f32100,0x625000f344b8) freed by thread T0 here: #0 0x7fbd770b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf) #1 0x55c0a5f1a02b in crimson::osd::PG::~PG() (/usr/bin/ceph-osd+0x1f62902b) previously allocated by thread T0 here: #0 0x7fbd770b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367) Reactor stalled for 203 ms on shard 0. Backtrace: 0x45d5d 0x2c67ec1e 0x2c67ffcc 0x2c68151a 0x2c68189e 0x2c6819e8 0x2c681e3e 0x54daf 0xc4f5e 0xc53da 0xc54b7 0xc5a38 0xc4612 0xcd073 0x29fcf07ea36f 0x29fcf07ea597 0x29fcf07f8d34 0x29fcf07ea18b 0x29fcf07ebfea 0xd6280 0x2f11c 0x32813 0xbd907 0xbd194 0xbdfda 0x1f6d5172 0x1fd5c708 0x1fd5cdbf 0x1fd5dcdc 0x1e62f198 0x1e76a046 0x1e76e196 0x1e76ebfe 0x1fc521dc 0x2d255050 0x2d39e3d1 0x1f87ad03 0x1f66a03b 0x1fbf8c45 0x1fcdb5ac 0x1fcdc712 0x2c62295b 0x2c6bc51c 0x2c8da55e 0x2c8dc281 0x2c3354f2 0x2c3373fb 0x1eb826c8 0x3feaf 0x3ff5f 0x1e62ba44 kernel callstack: 0xffffffffffffff80 0xffffffff85c89a14 0xffffffff86865842 0xffffffff86a00b82 #1 0x55c0a6c62ca6 in auto crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}::operator()<std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) const (/usr/bin/ceph-osd+0x20371ca6) #2 0x55c0a6c63a9c in auto seastar::futurize_invoke<crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > > >(crimson::osd::ShardServices::make_pg(crimson::local_shared_foreign_ptr<boost::local_shared_ptr<OSDMap const> >, spg_t, bool)::{lambda(auto:1&&)ceph#3}&, std::tuple<seastar::future<std::tuple<pg_pool_t, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > > > > >, seastar::future<boost::intrusive_ptr<crimson::os::FuturizedCollection> > >&&) (/usr/bin/ceph-osd+0x20372a9c) ceph#3 0x55c0b5c10b87 (/usr/bin/ceph-osd+0x2f31fb87) SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1f6d5172) in crimson::osd::operator<<(std::ostream&, crimson::osd::PG const&) ``` Signed-off-by: Matan Breizman <mbreizma@redhat.com>

``` DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - ShardServices::dispatch_context_transaction: empty transaction DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - do_peering_event ignoring epoch_sent: 37 epoch_requested: 37 RenewLease -- pg has reset DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - peering_event(id=33559971, detail=PeeringEvent(from=0 pgid=42.9 sent=37 requested=37 evt=epoch_sent: 37 epoch_requested: 37 RenewLease)): submitting ctx DEBUG 2023-11-14 15:23:50,871 [shard 2] osd - ShardServices::dispatch_context_transaction: empty transaction #0 0x5653c613c071 in seastar::shared_mutex::unlock() (/usr/bin/ceph-osd+0x1ed27071) DEBUG 2023-11-14 15:23:50,871 [shard 0] osd - snaptrim_event(id=10610, detail=SnapTrimEvent(pgid=16.1a snapid=a needs_pause=0)): interrupted crimson::common::actingset_changed (acting set changed) DEBUG 2023-11-14 15:23:50,927 [shard 0] osd - pg_advance_map(id=10742, detail=PGAdvanceMap(pg=24.2 from=46 to=48)): complete #1 0x5653c8670acf in auto seastar::futurize_invoke<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&>(crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::ExitBarrier<crimson::OrderedConcurrentPhaseT<crimson::osd::SnapTrimEvent::WaitSubop>::BlockingEvent::Trigger<crimson::osd::SnapTrimEvent> >::exit()::{lambda()#1}&) (/usr/bin/ceph-osd+0x2125bacf) #2 0x5653c8670e22 in _ZN7seastar20noncopyable_functionIFNS_6futureIvEEvEE17direct_vtable_forIZNS2_4thenIZN7crimson23OrderedConcurrentPhaseTINS7_3osd13SnapTrimEvent9WaitSubopEE11ExitBarrierINSC_13BlockingEvent7TriggerISA_EEE4exitEvEUlvE_S2_EET0_OT_EUlDpOT_E_E4callEPKS4_ (/usr/bin/ceph-osd+0x2125be22) ceph#3 0x5653c5d0b4b2 in auto seastar::internal::future_invoke<seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate>(seastar::noncopyable_function<seastar::future<void> ()>&, seastar::internal::monostate&&) (/usr/bin/ceph-osd+0x1e8f64b2) ceph#4 0x5653c5d3069f in void seastar::futurize<seastar::future<void> >::satisfy_with_result_of<seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}::operator()(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&) const::{lambda()#1}>(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&&) (/usr/bin/ceph-osd+0x1e91b69f) ceph#5 0x5653c5d4e09d in seastar::continuation<seastar::internal::promise_base_with_type<void>, seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void>::then_impl_nrvo<seastar::noncopyable_function<seastar::future<void> ()>, seastar::future<void> >(seastar::noncopyable_function<seastar::future<void> ()>&&)::{lambda(seastar::internal::promise_base_with_type<void>&&, seastar::noncopyable_function<seastar::future<void> ()>&, seastar::future_state<seastar::internal::monostate>&&)#1}, void>::run_and_dispose() (/usr/bin/ceph-osd+0x1e93909d) DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map ceph#6 0x5653d3bc3e2b in seastar::reactor::run_tasks(seastar::reactor::task_queue&) (/usr/bin/ceph-osd+0x2c7aee2b) DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_epoch 48 pg[36.0( empty local-lis/les=25/26 n=0 ec=25/25 lis/c=25/25 les/c/f=26/27/0 sis=25) [2,0] r=1 lpr=25 crt=0'0 mlcod 0'0 active PeeringState::activate_map : Not dirtying info: last_persisted is 28 while current is 48 DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - log is not dirty ceph#7 0x5653d3c61c42 in seastar::reactor::run_some_tasks() (/usr/bin/ceph-osd+0x2c84cc42) ceph#8 0x5653d3e84afa in seastar::reactor::do_run() (/usr/bin/ceph-osd+0x2ca6fafa) DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10706, detail=PGAdvanceMap(pg=17.17 from=46 to=48)): exit ceph#9 0x5653d3e88306 in seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}::operator()() const (/usr/bin/ceph-osd+0x2ca73306) DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10662, detail=PGAdvanceMap(pg=9.1 from=46 to=48)): complete ceph#10 0x5653d3e888b2 in std::_Function_handler<void (), seastar::smp::configure(seastar::smp_options const&, seastar::reactor_options const&)::{lambda()ceph#3}>::_M_invoke(std::_Any_data const&) (/usr/bin/ceph-osd+0x2ca738b2) DEBUG 2023-11-14 15:23:50,928 [shard 0] osd - pg_advance_map(id=10680, detail=PGAdvanceMap(pg=12.4 from=46 to=48)): sending pg temp ceph#11 0x5653d39775cf in seastar::posix_thread::start_routine(void*) (/usr/bin/ceph-osd+0x2c5625cf) DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): complete ceph#12 0x7f106029f801 in start_thread (/lib64/libc.so.6+0x9f801) ceph#13 0x7f106023f44f in __GI___clone3 (/lib64/libc.so.6+0x3f44f) DEBUG 2023-11-14 15:23:50,928 [shard 2] osd - pg_advance_map(id=33559841, detail=PGAdvanceMap(pg=18.9 from=46 to=48)): exit DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - pg_advance_map(id=33559863, detail=PGAdvanceMap(pg=21.e from=46 to=48)): complete 0x6150002c9a5c is located 92 bytes inside of 472-byte region [0x6150002c9a00,0x6150002c9bd8) DEBUG 2023-11-14 15:23:50,929 [shard 2] osd - do_peering_event handling epoch_sent: 19 epoch_requested: 19 RenewLease for pg: 18.9 freed by thread T1 here: #0 0x7f10628b73cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf) #1 0x5653c8794bff in crimson::osd::SnapTrimEvent::~SnapTrimEvent() (/usr/bin/ceph-osd+0x2137fbff) previously allocated by thread T1 here: #0 0x7f10628b6367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367) Thread T1 created by T0 here: #0 0x7f10628587d5 in pthread_create (/lib64/libasan.so.6+0x587d5) DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_epoch 48 pg[27.5( empty local-lis/les=0/0 n=0 ec=19/19 lis/c=0/0 les/c/f=0/0/0 sis=31) [] r=-1 lpr=31 pi=[19,31)/1 crt=0'0 mlcod 0'0 unknown NOTIFY PeeringState::activate_map : Not dirtying info: last_persisted is 31 while current is 48 #1 0x5653d3b89a69 in seastar::posix_thread::posix_thread(seastar::posix_thread::attr, std::function<void ()>) (/usr/bin/ceph-osd+0x2c774a69) DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - log is not dirty SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-osd+0x1ed27071) in seastar::shared_mutex::unlock() DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10821, detail=PGAdvanceMap(pg=48.3 from=46 to=48)): exit DEBUG 2023-11-14 15:23:50,934 [shard 0] osd - pg_advance_map(id=10733, detail=PGAdvanceMap(pg=22.11 from=46 to=48)): exit ``` Signed-off-by: Matan Breizman <mbreizma@redhat.com>

osd: Adding const to methods in OSDMap

576203f

Signed-off-by: Adam Crume <adamcrume@gmail.com>

liewegas closed this May 31, 2012

ddiss added a commit to ddiss/ceph that referenced this pull request Aug 6, 2015

Merge pull request ceph#11 from SUSE/osd_scsi_compare_and_write_same

4d5fb5e

OSD: extent compare and write same operations

chamdoo pushed a commit to chamdoo/ceph that referenced this pull request Nov 13, 2015

fix issue ceph#11

0142d38

URL: facebook/rocksdb#11

XinzeChi pushed a commit to XinzeChi/ceph that referenced this pull request Jan 29, 2016

Merge pull request ceph#11 from XinzeChi/wip-filestore-finisher

2e682ce

filestore: separete ondisk finisher with apply finisher Reviewed-by: Haomai Wang <haomai@xsky.com>

mathslinux added a commit to mathslinux/ceph that referenced this pull request Mar 21, 2016

Merge pull request ceph#11 from zhaochao/eayunobs-fix-user-quota-adju…

56d5153

…sting-on-bucket-removal rgw: user quota may not adjust on bucket removal

ifed01 pushed a commit to ifed01/ceph that referenced this pull request May 16, 2016

Merge pull request ceph#11 from ifed01/wip-bluestore-read-compress4

29f68ef

os/bluestore: Adds lacking methods in bluestore_compression_header_t …

lnsyyj mentioned this pull request Nov 10, 2016

rgw: fix Content-Type for S3 user create #11828

Closed

liewegas pushed a commit that referenced this pull request Dec 14, 2016

Merge pull request #11 from ceph/wip-7581

b727417

increase verbosity for OSDs for ceph-deploy tests

liewegas mentioned this pull request Dec 20, 2016

librados: AioCompletion + RGW #12377

Closed

dillaman mentioned this pull request Jan 23, 2017

common/ceph_crypto: warn + assert if crypto shutdown fails #12624

Closed

4 tasks

Abhishekvrshny pushed a commit to Abhishekvrshny/ceph that referenced this pull request Jul 13, 2017

Merge pull request ceph#11 from Flipkart/percentile_prashant

67f2ad6

Adding percentile

shangfufei mentioned this pull request Oct 10, 2017

msg/async: fix bug of data type conversion when uint64_t -> int -> uint64_t #18210

Merged

ivancich mentioned this pull request Nov 3, 2017

rgw: fix BZ 1500904, stale bucket index entry remains after obj delete #18709

Merged

theanalyst mentioned this pull request Nov 7, 2017

luminous: rgw: Stale bucket index entry remains after object deletion #18794

Merged

tianshan mentioned this pull request Dec 4, 2017

rgw: fix realm reload cause crash in stop rgw #19308

Closed

LenzGr pushed a commit to LenzGr/ceph that referenced this pull request Jan 24, 2018

Merge pull request ceph#11 from votdev/add_editorconfig

aef0343

mgr/dashboard_v2: Create proper text editor config files to maintain consistent coding …

sebastian-philipp pushed a commit to sebastian-philipp/ceph that referenced this pull request Nov 23, 2020

Merge pull request ceph#11 from adk3798/haproxy-prereq

ebc4031

warn if prerequisite for deploying haproxy and keepalived not met

tchaikov mentioned this pull request Jun 1, 2021

auth/KeyRing: always decode keying as plaintext #41631

Merged

ronen-fr mentioned this pull request Jul 27, 2021

osd/scrub: remove Scrubber sub-objects reliance on PG::gen_prefix() #42479

Merged

pritha-srivastava mentioned this pull request Nov 24, 2021

rgw multisite: replicate metadata for iam roles #43597

Merged

3 tasks

This was referenced Mar 23, 2022

cls/rgw : Add missing classes in < #include "cls/rgw/cls_rgw_types.h"> #45394

Merged

[DNM] Wip iqbal cls rgw classes #45538

Closed

liujiangang01 mentioned this pull request Oct 11, 2022

client: _readdir_cache_cb() may use the readdir_cache already clear #29526

Merged

3 tasks

dang mentioned this pull request Jun 2, 2023

Wip posix ordered #51519

Merged

14 tasks

tobias-urdin pushed a commit to tobias-urdin/ceph that referenced this pull request Aug 2, 2023

Merge pull request ceph#11 from andrewgaul/long-bucket-name-tests

d0e4dd0

Call get_new_bucket_name in long bucket name tests Reviewed-by: Yehuda Sadeh <yehuda@redhat.com>

joscollin mentioned this pull request Mar 19, 2024

cephfs_mirror: fix crash in update_fs_mirrors() #56148

Merged

14 tasks

arm7star mentioned this pull request Sep 12, 2024

test/crimson/test_messenger_thrash: fix a local variable scope issue #59746

Merged

14 tasks

mohit84 mentioned this pull request Apr 3, 2025

crimson/osd/pg_recovery: use OperationThrottler to throttle object pushes/pulls #62080

Merged

14 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

osd: Adding const to methods in OSDMap#11

osd: Adding const to methods in OSDMap#11
adamcrume wants to merge 1 commit intoceph:masterfrom
adamcrume:master

adamcrume commented May 31, 2012

Uh oh!

liewegas commented May 31, 2012

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

adamcrume commented May 31, 2012

Uh oh!

liewegas commented May 31, 2012

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants