Ceph » CephFS

We see a similar crash with quincy

27: FAILED ceph_assert(((mds->mds_lock).is_locked_by_me()))                                                                                                                   

 ceph version 17.2.8-4-gfdcded4f4fe (fdcded4f4fe652b62c46d2580aa45b936a5ab615) quincy (stable)                                                                                                                  
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11c) [0x7ff4fb421919]                                                                                                               
 2: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4fb421b20]                                                                                                                              
 3: (MDSContext::complete(int)+0xd1) [0x560ae1635f31]                                                                                                                                                           
 4: (Finisher::finisher_thread_entry()+0x5e1) [0x7ff4fb3b0359]                                                                                                                                                  
 5: (Finisher::FinisherThread::entry()+0xd) [0x560ae12c696b]                                                                                                                                                    
 6: (Thread::entry_wrapper()+0x3f) [0x7ff4fb3f2849]                                                                                                                                                             
 7: (Thread::_entry_func(void*)+0x9) [0x7ff4fb3f2861]                                                                                                                                                           
 8: /lib64/libc.so.6(+0x897e2) [0x7ff4fa2897e2]                                                                                                                                                                 
 9: /lib64/libc.so.6(+0x10e800) [0x7ff4fa30e800]                                                                                                                                                                

    -3> 2025-03-27T15:28:59.513+0100 7ff4f59f6640 10 monclient: handle_mon_command_ack 2 [{"prefix":"osd blocklist", "blocklistop":"add","addr":"188.185.19.81:0/2540936658"}]                                  
    -2> 2025-03-27T15:28:59.513+0100 7ff4f59f6640 10 monclient: _finish_command 2 = system:0 blocklisting 188.185.19.81:0/2540936658 until 2025-03-27T16:28:58.570465+0100 (3600 sec)                           
    -1> 2025-03-27T15:28:59.513+0100 7ff4f61f7640 10 monclient: _send_mon_message to mon.c at v2:188.185.19.81:40165/0                                                                                          
     0> 2025-03-27T15:28:59.515+0100 7ff4ef9ea640 -1 *** Caught signal (Aborted) **                                                                                                                             
 in thread 7ff4ef9ea640 thread_name:MR_Finisher                                                                                                                                                                 

 ceph version 17.2.8-4-gfdcded4f4fe (fdcded4f4fe652b62c46d2580aa45b936a5ab615) quincy (stable)                                                                                                                  
 1: /lib64/libc.so.6(+0x3e730) [0x7ff4fa23e730]
 2: /lib64/libc.so.6(+0x8b52c) [0x7ff4fa28b52c]
 3: raise()
 4: abort()
 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x263) [0x7ff4fb421a60]
 6: (ceph::register_assert_context(ceph::common::CephContext*)+0) [0x7ff4fb421b20]
 7: (MDSContext::complete(int)+0xd1) [0x560ae1635f31]
 8: (Finisher::finisher_thread_entry()+0x5e1) [0x7ff4fb3b0359]
 9: (Finisher::FinisherThread::entry()+0xd) [0x560ae12c696b]
 10: (Thread::entry_wrapper()+0x3f) [0x7ff4fb3f2849] 
 11: (Thread::_entry_func(void*)+0x9) [0x7ff4fb3f2861]
 12: /lib64/libc.so.6(+0x897e2) [0x7ff4fa2897e2]
 13: /lib64/libc.so.6(+0x10e800) [0x7ff4fa30e800]
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Abhishek Lekshmanan wrote in #note-3:

We see a similar crash with quincy

[...]

created https://tracker.ceph.com/issues/70769

The mutex crash likely prevents us from seeing my reported issue with BatchGetAttr as the mutex assert indicates one more potential issue where the MDSContext expects the MDS lock which is not held by the caller! However the BatchGetAttr issue clearly causes nullptr MDRequest objects which also might explain other crashes... potential PR (only for the batch issue): https://github.com/ceph/ceph/pull/62630

okay, so this issue is with the uninline data path during scrubbing
the uniniline code attempts a rdlock_path_pin_ref() on an inode while attempting to uninline it ... this is essentially bad code on my part
rdlock_path_pin_ref() attempts to do the locking and leads to a C_MDS_RetryRequest being attempted
But, when the code path reaches Server::dispatch_client_request() the mdr->client_request field is null

in short, since we know which inode we need to uninline, we should just use it