@kchheda3 @yuvalif this call is showing up in lifecycle crashes (see https://tracker.ceph.com/issues/64571#note-10). is it possible that res.bucket is null here?

it is possible. this ctor: https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_sal_rados.h#L736
does not initialize the bucket

it is possible. this ctor: https://github.com/ceph/ceph/blob/main/src/rgw/driver/rados/rgw_sal_rados.h#L736 does not initialize the bucket

but @yuvalif that ctor is for s3 requests, hence it has req_state and the code is checking if (!res.s), so that ctor will not be invoked.
@cbodley looking at all the calls for notification for LC which are inside this rgw_lc.cc, does not look like anywhere bucket is nullptr.

@cbodley looking at this crash which is non-notification one, it has the same later call trace which points to issue while reading the bucket instance info read_bucket_instance_info

wasn't the en2end tracing regression in LC because of the infinite recurssion?

ok, we finally have a root cause for this in https://tracker.ceph.com/issues/64571#note-33:

The cause for this issue seems to be due to multiple LC worker threads updating the same bucket handle, which is not thread-safe. When another load_bucket() call was added in remove_expirted_obj as part of #55657, multiple threads could update and overwrite the same ceph::buffer::list stored in bucket->attrs resulting in corruption at times.

we should probably keep the fix in PR #56712 regardless. if calling load_bucket() could be avoided in this case, we shoudl avoid it

we've got a couple more crashes from notification and LC setup together and both have different call stack but both fail in attrs decoding
this one is why decoding the bucket notification attrs during LC processing.
0x00007f6233140a9f in raise () from /lib64/libc.so.6 #1 0x00007f6233113ece in abort () from /lib64/libc.so.6 #2 0x00007f62373d89b6 in ceph::__ceph_assert_fail(char const*, char const*, int, char const*) () from /usr/lib64/ceph/libceph-common.so.2 #3 0x00007f62373d8b22 in ceph::__ceph_assert_fail(ceph::assert_data const&) () from /usr/lib64/ceph/libceph-common.so.2 #4 0x00007f62384659ba in ceph::buffer::v15_2_0::ptr::copy_out(unsigned int, unsigned int, char*) const () from /lib64/librados.so.2 #5 0x00007f623846a420 in ceph::buffer::v15_2_0::list::iterator_impl<true>::copy(unsigned int, char*) () from /lib64/librados.so.2 #6 0x00005590bfe34293 in ceph::decode(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, ceph::buffer::v15_2_0::list::iterator_impl<true>&) () #7 0x00005590c01131e6 in rgw_pubsub_dest::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) () #8 0x00005590c01135bf in rgw_pubsub_topic::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) () #9 0x00005590c011666e in rgw_pubsub_topic_filter::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) () #10 0x00005590c011871f in rgw_pubsub_bucket_topics::decode(ceph::buffer::v15_2_0::list::iterator_impl<true>&) () #11 0x00005590c02cead8 in get_bucket_notifications(DoutPrefixProvider const*, rgw::sal::Bucket*, rgw_pubsub_bucket_topics&) () #12 0x00005590c00baeda in rgw::notify::publish_reserve(DoutPrefixProvider const*, std::vector<rgw::notify::EventType, std::allocator<rgw::notify::EventType> > const&, rgw::notify::reservation_t&, RGWObjTags const*) () #13 0x00005590c0284f3e in remove_expired_obj(DoutPrefixProvider const*, lc_op_ctx&, bool, std::vector<rgw::notify::EventType, std::allocator<rgw::notify::EventType> > const&) () #14 0x00005590c029eea9 in LCOpAction_CurrentExpiration::process(lc_op_ctx&) () #15 0x00005590c0286acf in LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider const*, WorkQ*) () #16 0x00005590c0290063 in RGWLC::bucket_lc_process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, RGWLC::LCWorker*, long, bool)::{lambda(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&)#8}::operator()(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&) const () #17 0x00005590c029a0ad in WorkQ::entry() () #18 0x00007f62341031cf in start_thread () from /lib64/libpthread.so.0 #19 0x00007f623312bdd3 in clone () from /lib64/libc.so.6

and this one is during get_obj_state

#0 0x00007f09afaded6a in tc_newarray () from /lib64/libtcmalloc.so.4 #1 0x00007f09b306e755 in ceph::buffer::v15_2_0::ptr_node::cloner::operator()(ceph::buffer::v15_2_0::ptr_node const&) () from /lib64/librados.so.2 #2 0x0000561cb3d4a973 in rgw_filter_attrset(std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*) () #3 0x0000561cb3ef8fb2 in RGWRados::raw_obj_stat(DoutPrefixProvider const*, rgw_raw_obj&, unsigned long*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, unsigned long*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, ceph::buffer::v15_2_0::list*, RGWObjVersionTracker*, optional_yield) () #4 0x0000561cb3f17e04 in RGWRados::get_obj_state_impl(DoutPrefixProvider const*, RGWObjectCtx*, RGWBucketInfo&, rgw_obj const&, RGWObjState**, RGWObjManifest**, bool, optional_yield, bool) () #5 0x0000561cb3f19907 in RGWRados::get_obj_state(DoutPrefixProvider const*, RGWObjectCtx*, RGWBucketInfo&, rgw_obj const&, RGWObjState**, RGWObjManifest**, bool, optional_yield, bool) () #6 0x0000561cb3f1a7f7 in RGWRados::Object::get_state(DoutPrefixProvider const*, RGWObjState**, RGWObjManifest**, bool, optional_yield, bool) () #7 0x0000561cb3f246fa in RGWRados::Object::Delete::delete_obj(optional_yield, DoutPrefixProvider const*) () #8 0x0000561cb3ce23d2 in rgw::sal::RadosObject::RadosDeleteOp::delete_obj(DoutPrefixProvider const*, optional_yield) () #9 0x0000561cb3e7df5d in remove_expired_obj(DoutPrefixProvider const*, lc_op_ctx&, bool, std::vector<rgw::notify::EventType, std::allocator<rgw::notify::EventType> > const&) () #10 0x0000561cb3e97ea9 in LCOpAction_CurrentExpiration::process(lc_op_ctx&) () #11 0x0000561cb3e7facf in LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider const*, WorkQ*) () #12 0x0000561cb3e89063 in RGWLC::bucket_lc_process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, RGWLC::LCWorker*, long, bool)::{lambda(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&)#8}::operator()(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&) const () #13 0x0000561cb3e930ad in WorkQ::entry() ()

@cbodley you think this also could be due to multiple LC worker threads updating the bucket->handle(attrs) ??

@cbodley you think this also could be due to multiple LC worker threads updating the bucket->handle(attrs) ??

@kchheda3 assuming you haven't applied the fix from #56712, that sounds like a plausible explanation for both

@cbodley you think this also could be due to multiple LC worker threads updating the bucket->handle(attrs) ??

@kchheda3 assuming you haven't applied the fix from #56712, that sounds like a plausible explanation for both

yes i haven't applied that band-aid, but my understanding was with that band-aid the crash that it was fixing was different one load_bucket()->read_bucket_instance_info().. i guess this one
#4 0x000055dc73b9c3b0 in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_Reuse_or_alloc_node::operator()<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> const&>(std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> const&) () #5 0x000055dc73b9c628 in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_M_copy<false, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >*, std::_Rb_tree_node_base*, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_Reuse_or_alloc_node&) [clone .isra.0] () #6 0x000055dc73b9c65d in std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >* std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_M_copy<false, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_Reuse_or_alloc_node>(std::_Rb_tree_node<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >*, std::_Rb_tree_node_base*, std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::_Reuse_or_alloc_node&) [clone .isra.0] () #7 0x000055dc73b9ca65 in std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >::operator=(std::_Rb_tree<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list>, std::_Select1st<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > > const&) () #8 0x000055dc74082ff2 in RGWSI_Bucket_SObj::read_bucket_instance_info(ptr_wrapper<RGWSI_MetaBackend::Context, 4>&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, RGWBucketInfo*, std::chrono::time_point<ceph::real_clock, std::chrono::duration<unsigned long, std::ratio<1l, 1000000000l> > >*, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, ceph::buffer::v15_2_0::list, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, ceph::buffer::v15_2_0::list> > >*, optional_yield, DoutPrefixProvider const*, rgw_cache_entry_info*, boost::optional<obj_version>) () #9 0x000055dc7422c3cc in std::_Function_handler<int (ptr_wrapper<RGWSI_MetaBackend::Context, 4>&), RGWBucketCtl::read_bucket_instance_info(rgw_bucket const&, RGWBucketInfo*, optional_yield, DoutPrefixProvider const*, RGWBucketCtl::BucketInstance::GetParams const&)::{lambda(ptr_wrapper<RGWSI_MetaBackend::Context, 4>&)#1}>::_M_invoke(std::_Any_data const&, ptr_wrapper<RGWSI_MetaBackend::Context, 4>&) () #10 0x000055dc74244cb4 in std::_Function_handler<int (RGWSI_MetaBackend_Handler::Op*), RGWBucketInstanceMetadataHandler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (ptr_wrapper<RGWSI_MetaBackend::Context, 4>&)>)::{lambda(RGWSI_MetaBackend_Handler::Op*)#1}>::_M_invoke(std::_Any_data const&, RGWSI_MetaBackend_Handler::Op*&&) () #11 0x000055dc740a71de in std::_Function_handler<int (RGWSI_MetaBackend::Context*), RGWSI_MetaBackend_Handler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend_Handler::Op*)>)::{lambda(RGWSI_MetaBackend::Context*)#1}>::_M_invoke(std::_Any_data const&, RGWSI_MetaBackend::Context*&&) () #12 0x000055dc740a897f in RGWSI_MetaBackend_SObj::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend::Context*)>) () #13 0x000055dc740a7517 in RGWSI_MetaBackend_Handler::call(std::optional<std::variant<RGWSI_MetaBackend_CtxParams_SObj> >, std::function<int (RGWSI_MetaBackend_Handler::Op*)>) () #14 0x000055dc742317ab in RGWBucketCtl::read_bucket_instance_info(rgw_bucket const&, RGWBucketInfo*, optional_yield, DoutPrefixProvider const*, RGWBucketCtl::BucketInstance::GetParams const&) () #15 0x000055dc73f6ead5 in rgw::sal::RadosBucket::load_bucket(DoutPrefixProvider const*, optional_yield, bool) () #16 0x000055dc73f43bdd in rgw::notify::publish_reserve(DoutPrefixProvider const*, std::vector<rgw::notify::EventType, std::allocator<rgw::notify::EventType> > const&, rgw::notify::reservation_t&, RGWObjTags const*) () #17 0x000055dc7410cf3e in remove_expired_obj(DoutPrefixProvider const*, lc_op_ctx&, bool, std::vector<rgw::notify::EventType, std::allocator<rgw::notify::EventType> > const&) () #18 0x000055dc74126ea9 in LCOpAction_CurrentExpiration::process(lc_op_ctx&) () #19 0x000055dc7410eacf in LCOpRule::process(rgw_bucket_dir_entry&, DoutPrefixProvider const*, WorkQ*) () #20 0x000055dc74118063 in RGWLC::bucket_lc_process(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, RGWLC::LCWorker*, long, bool)::{lambda(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&)#8}::operator()(RGWLC::LCWorker*, WorkQ*, boost::variant<void*, std::tuple<LCOpRule, rgw_bucket_dir_entry>, std::tuple<lc_op, rgw_bucket_dir_entry>, rgw_bucket_dir_entry>&) const ()