Project

General

Profile

Actions

Bug #74960

closed

orch deployment: OSD::_write_key_meta aborts

Added by Matan Breizman 27 days ago. Updated 23 days ago.

Status:
Resolved
Priority:
High
Category:
Deployment
Target version:
-
% Done:

0%

Source:
Community (user)
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Fixed In:
v20.3.0-5417-g256b93633d
Released In:
Upkeep Timestamp:
2026-02-20T06:12:40+00:00

Description

Reported in Slack: https://ceph-storage.slack.com/archives/C05N7Q52DTL/p1770915641022979?thread_ts=1768340878.624389&cid=C05N7Q52DTL
Deployed via:

ceph orch apply osd --all-available-devices --method raw --osd_type crimson

bootstrap process looks ok:
Feb 13 03:24:34 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: INFO  2026-02-13 03:24:34,905 [shard 0:main] osd - get_early_config: set --thread-affinity 0 --smp 6
Feb 13 03:24:35 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: INFO   osd - OSD::main: early config parsed successfully

The following backtrace is reported:

Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: Aborting Got SIGABRT on shard 0 - Stopping all shardsGot SIGABRT on shard 0
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: Backtrace:
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:   0# 0x0000703CCB46903C in /lib64/libc.so.6
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  1# raise in /lib64/libc.so.6
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  2# abort in /lib64/libc.so.6
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  3# ceph::__ceph_abortf(char const*, int, char const*, char const*, ...) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  4# fmt::v11::report_error(char const*) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  5# crimson::osd::OSD::_write_key_meta(crimson::os::FuturizedStore&) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  6# void fmt::v11::detail::value<fmt::v11::context>::format_custom<MOSDBeacon, fmt::v11::formatter<MOSDBeacon, char, void> >(void*, fmt::v11::parse_context<char>&, fmt::v11::context&) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  7# void fmt::v11::detail::value<fmt::v11::context>::format_custom<MOSDBeacon, fmt::v11::formatter<MOSDBeacon, char, void> >(void*, fmt::v11::parse_context<char>&, fmt::v11::context&) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  8# _Unwind_RaiseException in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]:  9# _Unwind_RaiseException in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 10# _Unwind_RaiseException in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 11# _Unwind_RaiseException in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 12# google_breakpad::WriteMinidump(char const*, std::__cxx11::list<google_breakpad::MappingEntry, std::allocator<google_breakpad::MappingEntry> > const&, std::__cxx11::list<google_breakpad::AppMemory, std::allocator<google_breakpad::AppMemory> > const&, google_breakpad::LinuxDumper*) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 13# google_breakpad::WriteMinidump(char const*, std::__cxx11::list<google_breakpad::MappingEntry, std::allocator<google_breakpad::MappingEntry> > const&, std::__cxx11::list<google_breakpad::AppMemory, std::allocator<google_breakpad::AppMemory> > const&, google_breakpad::LinuxDumper*) in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 14# main in /usr/bin/ceph-osd-crimson
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 15# 0x0000703CCB406610 in /lib64/libc.so.6
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 16# __libc_start_main in /lib64/libc.so.6
Feb 13 03:24:36 data1.r1.hss1-ait3.laxb.gaikai.org bash[1971321]: 17# _start in /usr/bin/ceph-osd-crimson

Next steps:
  • I've suggested to enable OSD logs for now.
  • Possibly, this could relate to the face that Ubuntu is the distro being used. We have only tested with Centos (Rocky next) so far.
Actions #1

Updated by Matan Breizman 27 days ago

  • Assignee set to Shraddha Agrawal
Actions #2

Updated by Shraddha Agrawal 26 days ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 67374
Actions #3

Updated by Upkeep Bot 23 days ago

  • Status changed from Fix Under Review to Resolved
  • Merge Commit set to 256b93633db997d1af51e9c233cf1390c519a080
  • Fixed In set to v20.3.0-5417-g256b93633d
  • Upkeep Timestamp set to 2026-02-20T06:12:40+00:00
Actions

Also available in: Atom PDF