Project

General

Profile

Actions
Copy link

Bug #70723

closed

qa: AddressSanitizer reports heap-use-after-free in mds-log-replay thread

Added by Milind Changire 12 months ago. Updated 8 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Milind Changire
Category:
Correctness/Safety
Target version:
Ceph - v21.0.0
% Done:

0%

Source:
Development
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
MDS
Labels (FS):
Pull request ID:
62554
Tags (freeform):
Merge Commit:
742cb595719d5cc4c96edeefdff5d19009b9e506
Fixed In:
v20.3.0-1890-g742cb59571
Released In:
Upkeep Timestamp:
2025-07-25T11:15:02+00:00

Description

/teuthology/mchangir-2025-03-18_12:16:38-fs:workload-wip-mchangir-use-libc-for-segfault-main-debug-testing-default-smithi/8196888/teuthology.log

2025-03-18T13:00:53.869 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: =================================================================
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: ==2==ERROR: AddressSanitizer: heap-use-after-free on address 0x617000438208 at pc 0x55c9db8fba70 bp 0x7f1156cf6930 sp 0x7f1156cf6920
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: READ of size 8 at 0x617000438208 thread T54
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x55c9db8fba6f in std::_Rb_tree<dirfrag_t, dirfrag_t, std::_Identity<dirfrag_t>, std::less<dirfrag_t>, std::allocator<dirfrag_t> >::equal_range(dirfrag_t const&) (/usr/bin/ceph-mds+0xd16a6f)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x55c9db875e89 in std::_Rb_tree<dirfrag_t, dirfrag_t, std::_Identity<dirfrag_t>, std::less<dirfrag_t>, std::allocator<dirfrag_t> >::erase(dirfrag_t const&) (/usr/bin/ceph-mds+0xc90e89)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x55c9db78c116 in MDCache::finish_uncommitted_fragment(dirfrag_t, int) (/usr/bin/ceph-mds+0xba7116)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x55c9db13f5aa in EFragment::replay(MDSRank*) (/usr/bin/ceph-mds+0x55a5aa)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x55c9dc0158d8 in MDLog::_replay_thread() (/usr/bin/ceph-mds+0x14308d8)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9dbfd456a in MDLog::ReplayThread::entry() (/usr/bin/ceph-mds+0x13ef56a)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.870 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #8 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #9 0x7f116c1e112f in __GI___clone3 (/lib64/libc.so.6+0x10f12f)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: 0x617000438208 is located 520 bytes inside of 736-byte region [0x617000438000,0x6170004382e0)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: freed by thread T27 here:
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f3a93cf in operator delete(void*, unsigned long) (/lib64/libasan.so.6+0xb73cf)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x55c9dc00c20d in MDLog::standby_trim_segments() (/usr/bin/ceph-mds+0x142720d)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x55c9daf182fe in MDSRank::_standby_replay_restart_finish(int, unsigned long) (/usr/bin/ceph-mds+0x3332fe)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x55c9daf184d9 in MDSRank::C_MDS_StandbyReplayRestartFinish::finish(int) (/usr/bin/ceph-mds+0x3334d9)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x55c9dbfc1e27 in MDSContext::complete(int) (/usr/bin/ceph-mds+0x13dce27)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9dbfe0efd in MDSIOContextBase::complete(int) (/usr/bin/ceph-mds+0x13fbefd)
2025-03-18T13:00:53.871 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x7f116cb2e92a in Finisher::finisher_thread_entry() (/usr/lib64/ceph/libceph-common.so.2+0x4ee92a)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x7f116cb308ec in Finisher::FinisherThread::entry() (/usr/lib64/ceph/libceph-common.so.2+0x4f08ec)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #8 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #9 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #10 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: previously allocated by thread T50 here:
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f3a8367 in operator new(unsigned long) (/lib64/libasan.so.6+0xb6367)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x55c9dc01203c in MDLog::_replay_thread() (/usr/bin/ceph-mds+0x142d03c)
2025-03-18T13:00:53.875 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x55c9dbfd456a in MDLog::ReplayThread::entry() (/usr/bin/ceph-mds+0x13ef56a)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Thread T54 created by T27 here:
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f34a7d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x7f116cc9b7e6 in Thread::try_create(unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x65b7e6)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x7f116ccabe49 in Thread::create(char const*, unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x66be49)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x55c9dbff1ea4 in MDLog::replay(MDSContext*) (/usr/bin/ceph-mds+0x140cea4)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x55c9daf15022 in MDSRank::boot_start(MDSRank::BootStep, int) (/usr/bin/ceph-mds+0x330022)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9daf18311 in MDSRank::_standby_replay_restart_finish(int, unsigned long) (/usr/bin/ceph-mds+0x333311)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x55c9daf184d9 in MDSRank::C_MDS_StandbyReplayRestartFinish::finish(int) (/usr/bin/ceph-mds+0x3334d9)
2025-03-18T13:00:53.876 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x55c9dbfc1e27 in MDSContext::complete(int) (/usr/bin/ceph-mds+0x13dce27)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #8 0x55c9dbfe0efd in MDSIOContextBase::complete(int) (/usr/bin/ceph-mds+0x13fbefd)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #9 0x7f116cb2e92a in Finisher::finisher_thread_entry() (/usr/lib64/ceph/libceph-common.so.2+0x4ee92a)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #10 0x7f116cb308ec in Finisher::FinisherThread::entry() (/usr/lib64/ceph/libceph-common.so.2+0x4f08ec)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #11 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #12 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #13 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Thread T27 created by T14 here:
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f34a7d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x7f116cc9b7e6 in Thread::try_create(unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x65b7e6)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x7f116ccabe49 in Thread::create(char const*, unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x66be49)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x7f116cb22b15 in Finisher::start() (/usr/lib64/ceph/libceph-common.so.2+0x4e2b15)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x55c9daedbd95 in MDSRankDispatcher::init() (/usr/bin/ceph-mds+0x2f6d95)
2025-03-18T13:00:53.877 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9dae84a41 in MDSDaemon::handle_mds_map(boost::intrusive_ptr<MMDSMap const> const&) (/usr/bin/ceph-mds+0x29fa41)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x55c9dae8639e in MDSDaemon::handle_core_message(boost::intrusive_ptr<Message const> const&) (/usr/bin/ceph-mds+0x2a139e)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x55c9dae8841f in MDSDaemon::ms_dispatch2(boost::intrusive_ptr<Message> const&) (/usr/bin/ceph-mds+0x2a341f)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #8 0x7f116d1fbf34 in Messenger::ms_deliver_dispatch(boost::intrusive_ptr<Message> const&) (/usr/lib64/ceph/libceph-common.so.2+0xbbbf34)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #9 0x7f116d203311 in DispatchQueue::entry() (/usr/lib64/ceph/libceph-common.so.2+0xbc3311)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #10 0x7f116d648a48 in DispatchQueue::DispatchThread::entry() (/usr/lib64/ceph/libceph-common.so.2+0x1008a48)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #11 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #12 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #13 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Thread T14 created by T0 here:
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f34a7d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x7f116cc9b7e6 in Thread::try_create(unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x65b7e6)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x7f116ccabe49 in Thread::create(char const*, unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x66be49)
2025-03-18T13:00:53.878 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x7f116d1f6a20 in DispatchQueue::start() (/usr/lib64/ceph/libceph-common.so.2+0xbb6a20)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x7f116d64ec7a in AsyncMessenger::ready() (/usr/lib64/ceph/libceph-common.so.2+0x100ec7a)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9dae68dd1 in MDSDaemon::init() (/usr/bin/ceph-mds+0x283dd1)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x55c9dae3bf92 in main (/usr/bin/ceph-mds+0x256f92)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x7f116c0fb5cf in __libc_start_call_main (/lib64/libc.so.6+0x295cf)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Thread T50 created by T27 here:
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #0 0x7f116f34a7d5 in pthread_create (/lib64/libasan.so.6+0x587d5)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #1 0x7f116cc9b7e6 in Thread::try_create(unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x65b7e6)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #2 0x7f116ccabe49 in Thread::create(char const*, unsigned long) (/usr/lib64/ceph/libceph-common.so.2+0x66be49)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #3 0x55c9dbff1ea4 in MDLog::replay(MDSContext*) (/usr/bin/ceph-mds+0x140cea4)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #4 0x55c9daf15022 in MDSRank::boot_start(MDSRank::BootStep, int) (/usr/bin/ceph-mds+0x330022)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #5 0x55c9daf18311 in MDSRank::_standby_replay_restart_finish(int, unsigned long) (/usr/bin/ceph-mds+0x333311)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #6 0x55c9daf184d9 in MDSRank::C_MDS_StandbyReplayRestartFinish::finish(int) (/usr/bin/ceph-mds+0x3334d9)
2025-03-18T13:00:53.879 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #7 0x55c9dbfc1e27 in MDSContext::complete(int) (/usr/bin/ceph-mds+0x13dce27)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #8 0x55c9dbfe0efd in MDSIOContextBase::complete(int) (/usr/bin/ceph-mds+0x13fbefd)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #9 0x7f116cb2e92a in Finisher::finisher_thread_entry() (/usr/lib64/ceph/libceph-common.so.2+0x4ee92a)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #10 0x7f116cb308ec in Finisher::FinisherThread::entry() (/usr/lib64/ceph/libceph-common.so.2+0x4f08ec)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #11 0x7f116cc9b614 in Thread::entry_wrapper() (/usr/lib64/ceph/libceph-common.so.2+0x65b614)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #12 0x7f116cc9b650 in Thread::_entry_func(void*) (/usr/lib64/ceph/libceph-common.so.2+0x65b650)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:     #13 0x7f116c15c0c9 in start_thread (/lib64/libc.so.6+0x8a0c9)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: SUMMARY: AddressSanitizer: heap-use-after-free (/usr/bin/ceph-mds+0xd16a6f) in std::_Rb_tree<dirfrag_t, dirfrag_t, std::_Identity<dirfrag_t>, std::less<dirfrag_t>, std::allocator<dirfrag_t> >::equal_range(dirfrag_t const&)
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Shadow bytes around the buggy address:
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007eff0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f010: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.880 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f020: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f030: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: =>0x0c2e8007f040: fd[fd]fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f050: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f060: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f070: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f080: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   0x0c2e8007f090: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: Shadow byte legend (one shadow byte represents 8 application bytes):
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Addressable:           00
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Partially addressable: 01 02 03 04 05 06 07
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Heap left redzone:       fa
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Freed heap region:       fd
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Stack left redzone:      f1
2025-03-18T13:00:53.881 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Stack mid redzone:       f2
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Stack right redzone:     f3
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Stack after return:      f5
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Stack use after scope:   f8
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Global redzone:          f9
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Global init order:       f6
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Poisoned by user:        f7
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Container overflow:      fc
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Array cookie:            ac
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Intra object redzone:    bb
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   ASan internal:           fe
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Left alloca redzone:     ca
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Right alloca redzone:    cb
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]:   Shadow gap:              cc
2025-03-18T13:00:53.882 INFO:journalctl@ceph.mds.c.smithi062.stdout:Mar 18 13:00:53 smithi062 ceph-feab2b76-03f5-11f0-bb95-bd4984dce30f-mds-c[73467]: ==2==ABORTING

This points to mismanaged references to LogSegment objects.

Related issues 2 (1 open — 1 closed)

Related to CephFS - Bug #69953: mds: segmentation faults in recent QAPending BackportMahesh Mohan

Actions
Related to CephFS - Bug #54741: crash: MDSTableClient::got_journaled_ack(unsigned long)ResolvedVenky Shankar

Actions
Actions
Copy link
 #1

Updated by Milind Changire 12 months ago

  • Pull request ID set to 62554
Actions
Copy link
 #2

Updated by Venky Shankar 12 months ago

  • Category set to Correctness/Safety
  • Status changed from New to Triaged
  • Assignee set to Milind Changire
  • Target version set to v20.0.0
  • Source set to Development
  • Component(FS) MDS added
Actions
Copy link
 #3

Updated by Milind Changire 12 months ago

  • Related to Bug #69953: mds: segmentation faults in recent QA added
Actions
Copy link
 #4

Updated by Venky Shankar 11 months ago

  • Status changed from Triaged to Fix Under Review
Actions
Copy link
 #5

Updated by Venky Shankar 8 months ago

  • Status changed from Fix Under Review to Resolved
  • Target version changed from v20.0.0 to v21.0.0

Let's bake it in main branch.

Actions
Copy link
 #6

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to 742cb595719d5cc4c96edeefdff5d19009b9e506
  • Fixed In set to v20.3.0-1890-g742cb59571
  • Upkeep Timestamp set to 2025-07-25T11:15:02+00:00
Actions
Copy link
 #7

Updated by Venky Shankar 7 months ago

  • Related to Bug #54741: crash: MDSTableClient::got_journaled_ack(unsigned long) added
Actions
Copy link

Also available in: Atom PDF