Project

General

Profile

Actions
Copy link

Bug #66852

closed

SeaStore Assertion `get_addr_type() == paddr_types_t::SEGMENT' failed.

Added by Yingxin Cheng over 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Category:
Seastore
Target version:
-
% Done:

0%

Source:
Q/A
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
60817
Tags (freeform):
Merge Commit:
739139b7de6e321b936cf9231ea535331ba9715b
Fixed In:
v19.3.0-6235-g739139b7de
Released In:
v20.2.0~1578
Upkeep Timestamp:
2025-11-01T01:17:14+00:00

Description

ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.0.0-4748-ge9ff42b2/rpm/el9/BUILD/ceph-19.0.0-4748-ge9ff42b2/src/crimson/os/seastore/seastore_types.h:792: crimson::os::seastore::seg_paddr_t& crimson::os::seastore::paddr_t::as_seg_paddr(): Assertion `get_addr_type()== paddr_types_t::SEGMENT' failed.
Aborting on shard 0.
Backtrace:
0# 0x00007FC1E588B94C in /lib64/libc.so.6
1# raise in /lib64/libc.so.6
2# abort in /lib64/libc.so.6
3# 0x00007FC1E582871B in /lib64/libc.so.6
4# 0x00007FC1E5837386 in /lib64/libc.so.6
5# crimson::os::seastore::paddr_t::as_seg_paddr() in ceph-osd
6# 0x000000000974CCA9 in ceph-osd
...
37# 0x0000000009748488 in ceph-osd                                                                                                                                                                                                                                                                                                                                                            38# seastar::noncopyable_function<crimson::interruptible::interruptible_future_detail<crimson::os::seastore::TransactionConflictCondition, crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<5> > >::_future<crimson::errorated_future_marker<std::__cxx11::list<boost::intrusive_ptr<crimson::os::seastore::CachedExtent>, std::allocator<boost::                  intrusive_ptr<crimson::os::seastore::CachedExtent> > > > > > (seastar::future<std::__cxx11::list<std::unique_ptr<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>, std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> > >, std::allocator<std::unique_ptr<crimson::os::seastore::                 PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>, std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> > > > > >&&)>::operator()(seastar::future<std::__cxx11::list<std::unique_ptr<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>, std::default_delete<crimson::os::           seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> > >, std::allocator<std::unique_ptr<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>, std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> > > > > >&&) const in ceph-osd
Actions
Copy link
 #2

Updated by Matan Breizman over 1 year ago

  • Category set to Seastore
  • Source set to Q/A
Actions
Copy link
 #3

Updated by Nitzan Mordechai over 1 year ago

/a/nmordech-2024-07-09_05:22:27-crimson-rados-wip-nitzan-wait-osd-admin-command-distro-default-smithi/7793441

Actions
Copy link
 #4

Updated by Yingxin Cheng over 1 year ago · Edited

  • Status changed from New to Need More Info

Change to Need-More-Info, the backtrace doesn't reveal where it is actually failing, there are many users of `as_seg_paddr()`, and it is possible that this assert failure is a side-effect from another problem.

Actions
Copy link
 #5

Updated by Yingxin Cheng over 1 year ago 

seastar::noncopyable_function<
  crimson::interruptible::interruptible_future_detail<
    crimson::os::seastore::TransactionConflictCondition,
    crimson::errorator<crimson::unthrowable_wrapper<std::error_code const&, crimson::ec<5> >
  >::_future<crimson::errorated_future_marker<std::__cxx11::list<
    boost::intrusive_ptr<crimson::os::seastore::CachedExtent>,
    std::allocator<boost::intrusive_ptr<crimson::os::seastore::CachedExtent> >
  > > >
> (seastar::future<std::__cxx11::list<
  std::unique_ptr<
    crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>,
    std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> >
  >,
  std::allocator<std::unique_ptr<
    crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>,
    std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> >
  >
> > >&&)>::operator()(seastar::future<std::__cxx11::list<
  std::unique_ptr<
    crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>,
    std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> >
  >,
  std::allocator<std::unique_ptr<
    crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t>,
    std::default_delete<crimson::os::seastore::PhysicalNodeMapping<unsigned long, crimson::os::seastore::paddr_t> >
  >
> > >&&) const in ceph-osd

Seems like somewhere in a lambda returning a list of PhysicalNodeMapping.

Actions
Copy link
 #10

Updated by Yingxin Cheng over 1 year ago

May happen during seastore read:

...
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] osd - handling op read on object 3:dbac0dea:::smithi05532027-302:head
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] seastore_t - 0x517000729600 trans.5806 Cache::create_transaction: created name=read, source=READ, is_weak=false
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] seastore - 0x517000729600 trans.5806 SeaStoreS::repeat_with_onode: read cid=3.b_head oid=#3:dbac0dea:::smithi05532027-302:head# ...
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] seastore_cache - 0x517000729600 trans.5806 Cache::get_root: root not on t -- CachedExtent(addr=0x515000067a00, type=ROOT, trans=0, pending_io=0, version=5,                            dirty_from_or_retired_at=jseq(sseq(3), paddr<Seg[Dev(0x0),0x15],0x605000>), modify_time=tp(2024-11-20 06:33:39), paddr=paddr<Dev(ROOT),0x0>, prior_paddr=nullopt, length=0x0, loaded=0x0, state=DIRTY, last_committed_crc=0, refcount=7, user_hint=Hint(NULL), rewrite_gen=GEN_NULL, root_block(lba_root_node=0x5160002da580, backref_root_node=0x5160006aaa80))
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] seastore_tm - 0x517000729600 trans.5806 TransactionManager::read_extent: L0xffff30f0e8261000 ONODE_BLOCK_STAGED ...
DEBUG 2024-11-20 06:35:58,163 [shard 0:main] seastore_tm - 0x517000729600 trans.5806 TransactionManager::get_pin: L0xffff30f0e8261000 ...
ceph-osd: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-6091-g80596e19/rpm/el9/BUILD/ceph-19.3.0-6091-g80596e19/src/crimson/os/seastore/seastore_types.h:793: crimson::os::seastore::seg_paddr_t& crimson::os::seastore::paddr_t::as_seg_paddr(): Assertion `get_addr_type() == paddr_types_t::SEGMENT' failed.
Aborting on shard 0.

https://pulpito.ceph.com/yingxin-2024-11-20_06:04:58-crimson-rados-ci-wip-yingxin-seastore-fine-grained-cache-8-distro-default-smithi/8001108/ osd 3

Actions
Copy link
 #11

Updated by MOHIT AGRAWAL over 1 year ago

The osd.1 is aborted for the job https://pulpito.ceph.com/moagrawa-2024-11-21_08:55:14-crimson-rados-wip-mohit-crimson-device_class-distro-crimson-smithi/8003305/

DEBUG 2024-11-21 10:18:18,807 [shard 1:main] osd - OSDSingletonState::dispatch_context_transaction: empty transaction
DEBUG 2024-11-21 10:18:18,807 [shard 1:main] seastore_t - TransactionManager::flush: H0x5070002a72d8 start
DEBUG 2024-11-21 10:18:18,823 [shard 1:main] seastore_journal - SegmentedJournal::flush: H0x5070002a72d8 flush ...
DEBUG 2024-11-21 10:18:18,823 [shard 1:main] seastore_journal - SegmentedJournal::flush: 
H0x5070002a72d8 flush done
DEBUG 2024-11-21 10:18:18,823 [shard 1:main] seastore_t - TransactionManager::flush: H0x5070002a72d8 completed
 0# 0x00007F70D428B99C in /lib64/libc.so.6
 1# raise in /lib64/libc.so.6 
 2# abort in /lib64/libc.so.6 
 3# 0x00007F70D422871B in /lib64/libc.so.6
 4# 0x00007F70D4237386 in /lib64/libc.so.6
 5# crimson::os::seastore::paddr_t::as_seg_paddr() in ceph-osd

Actions
Copy link
 #12

Updated by Yingxin Cheng over 1 year ago

Actions
Copy link
 #13

Updated by Yingxin Cheng over 1 year ago

10 times of tasks/readwrite cannot reproduce the issue after PR60817, will mark this as resolved.

Actions
Copy link
 #14

Updated by Yingxin Cheng over 1 year ago

  • Pull request ID set to 60817
Actions
Copy link
 #15

Updated by Yingxin Cheng over 1 year ago

  • Status changed from Need More Info to Resolved
Actions
Copy link
 #16

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to 739139b7de6e321b936cf9231ea535331ba9715b
  • Fixed In set to v19.3.0-6235-g739139b7de6
  • Upkeep Timestamp set to 2025-07-10T19:37:12+00:00
Actions
Copy link
 #17

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-6235-g739139b7de6 to v19.3.0-6235-g739139b7de
  • Upkeep Timestamp changed from 2025-07-10T19:37:12+00:00 to 2025-07-14T22:42:22+00:00
Actions
Copy link
 #18

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~1578
  • Upkeep Timestamp changed from 2025-07-14T22:42:22+00:00 to 2025-11-01T01:17:14+00:00
Actions
Copy link

Also available in: Atom PDF