Ceph » CephFS

There could be different back traces but generally this looks like a mess with open file descriptors.
Observed when there are multiple directories to sync and cephfs_mirror_max_concurrent_directory_syncs > 1 (32 has been configured in our case).

The issue started to appear after upgrade from v17.2.7 to 18.2.4 so apparently this is a refression.

The most common backtrace/assertion looks like:
Oct 31 10:37:36 ceph-mds03 cephfs-mirror[3155735]: /home/abuild/rpmbuild/BUILD/ceph-18.2.4/src/client/Inode.cc: In function 'bool Inode::put_open_ref(int)' thread 7efd387b5700 tim>
Oct 31 10:37:36 ceph-mds03 cephfs-mirror[3155735]: /home/abuild/rpmbuild/BUILD/ceph-18.2.4/src/client/Inode.cc: 175: FAILED ceph_assert(ref > 0)

Thread 76 "replayer-27" received signal SIGABRT, Aborted.
[Switching to Thread 0x7efd387b5700 (LWP 3155835)]
0x00007efd5f319d2b in raise () from target:/lib64/libc.so.6
(gdb) bt
#0  0x00007efd5f319d2b in raise () from target:/lib64/libc.so.6
#1  0x00007efd5f31b3e5 in abort () from target:/lib64/libc.so.6
#2  0x00007efd5f9defd1 in ceph::__ceph_assert_fail(char const*, char const*, int, char const*) () from target:/usr/lib64/ceph/libceph-common.so.2
#3  0x00007efd5f9df117 in ceph::__ceph_assert_fail(ceph::assert_data const&) () from target:/usr/lib64/ceph/libceph-common.so.2
#4  0x00007efd605cfc6e in Inode::put_open_ref(int) () from target:/usr/lib64/libcephfs.so.2
#5  0x00007efd605767ab in Client::_release_fh(Fh*) () from target:/usr/lib64/libcephfs.so.2
#6  0x00007efd60576b96 in Client::_close(int) () from target:/usr/lib64/libcephfs.so.2
#7  0x00007efd60577046 in Client::_closedir(dir_result_t*) () from target:/usr/lib64/libcephfs.so.2
#8  0x00007efd605774d2 in Client::closedir(dir_result_t*) () from target:/usr/lib64/libcephfs.so.2
#9  0x000056534c2f790d in cephfs::mirror::PeerReplayer::propagate_deleted_entries(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cephfs::mirror::PeerReplayer::FHandles const&) ()
#10 0x000056534c2f8adc in cephfs::mirror::PeerReplayer::do_synchronize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> const&, boost::optional<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> >) ()
#11 0x000056534c2fa361 in cephfs::mirror::PeerReplayer::synchronize(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> const&, boost::optional<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, unsigned long> >) ()
#12 0x000056534c2fb135 in cephfs::mirror::PeerReplayer::do_sync_snaps(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) ()
#13 0x000056534c2fc79e in cephfs::mirror::PeerReplayer::sync_snaps(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::unique_lock<std::mutex>&) ()
#14 0x000056534c2fd12d in cephfs::mirror::PeerReplayer::run(cephfs::mirror::PeerReplayer::SnapshotReplayerThread*) ()
#15 0x000056534c302a30 in cephfs::mirror::PeerReplayer::SnapshotReplayerThread::entry() ()
#16 0x00007efd5f7406ea in start_thread () from target:/lib64/libpthread.so.0
#17 0x00007efd5f3e758f in clone () from target:/lib64/libc.so.6