Ceph » bluestore

Syntax: ceph-objectstore-tool --data-path $PATH_TO_OSD --pgid $PG_ID $OBJECT get-attrs $KEY > $OBJECT_ATTRS_FILE_NAME
The command execution results in a core dump due to the BlueFS ENOSPC error on the OSD, but this command should have ideally worked without the crash

./bin/ceph-objectstore-tool --data-path dev/osd0 '["2.a",{"oid":"benchmark_data_o08_1248413_object12275","key":"","snapid":-2,"hash":3749470186,"max":0,"pool":2,"namespace":"","max":0}]' get-attr snapset

/home/jayaprakash/ceph/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range_F(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f7384e14900 time 2025-11-04T11:42:43.168107+0000
/home/jayaprakash/ceph/src/os/bluestore/BlueFS.cc: 3957: ceph_abort_msg("bluefs enospc")
 ceph version 20.3.0-3858-gd624082cb6a (d624082cb6af27cfa96b8953aef5b93802fe8316) tentacle (dev - Debug)
 1: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0xc0) [0x7f73861237f7]
 2: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x8b0) [0x5577358b3d1a]
 3: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xd8) [0x5577358b42cc]
 4: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0x133) [0x5577358d5de1]
 5: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x18) [0x5577358ecf38]
 6: ./bin/ceph-objectstore-tool(+0x19b37bf) [0x557735c787bf]
 7: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x45c) [0x557735cbc7aa]
 8: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0xa81) [0x557735cb9d8b]
 9: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1e6) [0x557735f9bbee]
 10: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x179) [0x557735f9ae4d]
 11: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x141) [0x557735f9ac9d]
 12: (rocksdb::BlockBasedTableBuilder::Flush()+0x1da) [0x557735f9ab4c]
 13: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x1c1) [0x557735f9a477]
 14: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*)+0xdd8) [0x557735eb7ec6]
 15: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x76b) [0x557735aeb53b]
 16: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0xeb4) [0x557735ae93ce]
 17: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1d3a) [0x557735ae66ce]
 18: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x744) [0x557735aed408]
 19: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x52) [0x557735aec078]
 20: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x986) [0x5577359c4036]
 21: (RocksDBStore::open(std::ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16) [0x5577359dc898]
 22: (BlueStore::_open_db(bool, bool, bool)+0x845) [0x557735811ac3]
 23: (BlueStore::_open_db_and_around(bool, bool)+0x67e) [0x5577358159a4]
 24: (BlueStore::_mount()+0x680) [0x55773581a24a]
 25: (BlueStore::mount()+0x9) [0x557735867ca7]
 26: main()
 27: /lib64/libc.so.6(+0x295d0) [0x7f7384c295d0]
 28: __libc_start_main()
 29: _start()
*** Caught signal (Aborted) **
 in thread 7f7384e14900 thread_name:ceph-objectstor
 ceph version 20.3.0-3858-gd624082cb6a (d624082cb6af27cfa96b8953aef5b93802fe8316) tentacle (dev - Debug)
 1: /lib64/libc.so.6(+0x3ea60) [0x7f7384c3ea60]
 2: /lib64/libc.so.6(+0x8c0fc) [0x7f7384c8c0fc]
 3: raise()
 4: abort()
 5: (ceph::__ceph_abort(char const*, int, char const*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x247) [0x7f738612397e]
 6: (BlueFS::_flush_range_F(BlueFS::FileWriter*, unsigned long, unsigned long)+0x8b0) [0x5577358b3d1a]
 7: (BlueFS::_flush_F(BlueFS::FileWriter*, bool, bool*)+0xd8) [0x5577358b42cc]
 8: (BlueFS::append_try_flush(BlueFS::FileWriter*, char const*, unsigned long)+0x133) [0x5577358d5de1]
 9: (BlueRocksWritableFile::Append(rocksdb::Slice const&)+0x18) [0x5577358ecf38]
 10: ./bin/ceph-objectstore-tool(+0x19b37bf) [0x557735c787bf]
 11: (rocksdb::WritableFileWriter::WriteBuffered(char const*, unsigned long, rocksdb::Env::IOPriority)+0x45c) [0x557735cbc7aa]
 12: (rocksdb::WritableFileWriter::Append(rocksdb::Slice const&, unsigned int, rocksdb::Env::IOPriority)+0xa81) [0x557735cb9d8b]
 13: (rocksdb::BlockBasedTableBuilder::WriteMaybeCompressedBlock(rocksdb::Slice const&, rocksdb::CompressionType, rocksdb::BlockHandle*, rocksdb::BlockType, rocksdb::Slice const*)+0x1e6) [0x557735f9bbee]
 14: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::Slice const&, rocksdb::BlockHandle*, rocksdb::BlockType)+0x179) [0x557735f9ae4d]
 15: (rocksdb::BlockBasedTableBuilder::WriteBlock(rocksdb::BlockBuilder*, rocksdb::BlockHandle*, rocksdb::BlockType)+0x141) [0x557735f9ac9d]
 16: (rocksdb::BlockBasedTableBuilder::Flush()+0x1da) [0x557735f9ab4c]
 17: (rocksdb::BlockBasedTableBuilder::Add(rocksdb::Slice const&, rocksdb::Slice const&)+0x1c1) [0x557735f9a477]
 18: (rocksdb::BuildTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, rocksdb::VersionSet*, rocksdb::ImmutableDBOptions const&, rocksdb::TableBuilderOptions const&, rocksdb::FileOptions const&, rocksdb::TableCache*, rocksdb::InternalIteratorBase<rocksdb::Slice>*, std::vector<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> >, std::allocator<std::unique_ptr<rocksdb::FragmentedRangeTombstoneIterator, std::default_delete<rocksdb::FragmentedRangeTombstoneIterator> > > >, rocksdb::FileMetaData*, std::vector<rocksdb::BlobFileAddition, std::allocator<rocksdb::BlobFileAddition> >*, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, unsigned long, rocksdb::SnapshotChecker*, bool, rocksdb::InternalStats*, rocksdb::IOStatus*, std::shared_ptr<rocksdb::IOTracer> const&, rocksdb::BlobFileCreationReason, rocksdb::SeqnoToTimeMapping const&, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, rocksdb::Env::WriteLifeTimeHint, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const*, rocksdb::BlobFileCompletionCallback*, unsigned long*, unsigned long*, unsigned long*)+0xdd8) [0x557735eb7ec6]
 19: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x76b) [0x557735aeb53b]
 20: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool, bool*, rocksdb::DBImpl::RecoveryContext*)+0xeb4) [0x557735ae93ce]
 21: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool, unsigned long*, rocksdb::DBImpl::RecoveryContext*)+0x1d3a) [0x557735ae66ce]
 22: (rocksdb::DBImpl::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**, bool, bool)+0x744) [0x557735aed408]
 23: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0x52) [0x557735aec078]
 24: (RocksDBStore::do_open(std::ostream&, bool, bool, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x986) [0x5577359c4036]
 25: (RocksDBStore::open(std::ostream&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x16) [0x5577359dc898]
 26: (BlueStore::_open_db(bool, bool, bool)+0x845) [0x557735811ac3]
 27: (BlueStore::_open_db_and_around(bool, bool)+0x67e) [0x5577358159a4]
 28: (BlueStore::_mount()+0x680) [0x55773581a24a]
 29: (BlueStore::mount()+0x9) [0x557735867ca7]
 30: main()
 31: /lib64/libc.so.6(+0x295d0) [0x7f7384c295d0]
Aborted (core dumped)

Expected Results :

./bin/ceph-objectstore-tool --data-path dev/osd0 '["2.a",{"oid":"benchmark_data_o08_1241794_object1755","key":"","snapid":-2,"hash":4092739562,"max":0,"pool":2,"namespace":"","max":0}]' get-attr snapset
Base64:AwIdAAAAAAAAAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAA=