Project

General

Profile

Actions
Copy link

Bug #67087

open

osd: assertion failure during multi-region EC reads

Added by Radoslaw Zarzynski over 1 year ago. Updated over 1 year ago.

Status:
Fix Under Review
Priority:
Normal
Assignee:
Radoslaw Zarzynski
Category:
-
Target version:
-
% Done:

0%

Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(RADOS):
Pull request ID:
58749
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:

Description

@Ilya Dryomov has reported the following crash in https://ceph-storage.slack.com/archives/C1HFZTW81/p1721376624460589:

2024-07-19T02:41:53.779 INFO:tasks.ceph.osd.0.smithi037.stderr:./src/osd/ECCommon.cc: In function 'static void ECCommon::ReadPipeline::get_min_want_to_read_shards(uint64_t, uint64_t, const ECUtil::stripe_info_t&, const std::vector<int>&, std::set<int>*)' thread 7f1caaa98640 time 2024-07-19T02:41:53.781585+0000
2024-07-19T02:41:53.780 INFO:tasks.ceph.osd.0.smithi037.stderr:./src/osd/ECCommon.cc: 336: FAILED ceph_assert(want_to_read->size() == sinfo.get_data_chunk_count())
2024-07-19T02:41:53.786 INFO:tasks.ceph.osd.0.smithi037.stderr: ceph version 19.0.0-5126-g01e1aa05 (01e1aa052c15e2a077a20de402d3ae763de4a30d) squid (dev)
2024-07-19T02:41:53.786 INFO:tasks.ceph.osd.0.smithi037.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x118) [0x55a0ce594810]
2024-07-19T02:41:53.786 INFO:tasks.ceph.osd.0.smithi037.stderr: 2: ceph-osd(+0x4009c7) [0x55a0ce5949c7]
2024-07-19T02:41:53.786 INFO:tasks.ceph.osd.0.smithi037.stderr: 3: ceph-osd(+0x3a709a) [0x55a0ce53b09a]
2024-07-19T02:41:53.786 INFO:tasks.ceph.osd.0.smithi037.stderr: 4: (ECCommon::ReadPipeline::get_min_want_to_read_shards(unsigned long, unsigned long, std::set<int, std::less<int>, std::allocator<int> >*)+0x56) [0x55a0ce881da6]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 5: (ECCommon::ReadPipeline::objects_read_and_reconstruct(std::map<hobject_t, std::__cxx11::list<ECCommon::ec_align_t, std::allocator<ECCommon::ec_align_t> >, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, std::__cxx11::list<ECCommon::ec_align_t, std::allocator<ECCommon::ec_align_t> > > > > const&, bool, std::unique_ptr<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&>, std::default_delete<GenContext<std::map<hobject_t, ECCommon::ec_extent_t, std::less<hobject_t>, std::allocator<std::pair<hobject_t const, ECCommon::ec_extent_t> > >&&> > >&&)+0x8f7) [0x55a0ce888a17]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 6: (ECBackend::objects_read_async(hobject_t const&, std::__cxx11::list<std::pair<ECCommon::ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> >, std::allocator<std::pair<ECCommon::ec_align_t, std::pair<ceph::buffer::v15_2_0::list*, Context*> > > > const&, Context*, bool)+0x5a0) [0x55a0cea98d70]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 7: (PrimaryLogPG::OpContext::start_async_reads(PrimaryLogPG*)+0x179) [0x55a0ce7cd669]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 8: (PrimaryLogPG::execute_ctx(PrimaryLogPG::OpContext*)+0x473) [0x55a0ce7f6153]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 9: (PrimaryLogPG::do_op(boost::intrusive_ptr<OpRequest>&)+0x30f3) [0x55a0ce7e1b83]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 10: (OSD::dequeue_op(boost::intrusive_ptr<PG>, boost::intrusive_ptr<OpRequest>, ThreadPool::TPHandle&)+0x197) [0x55a0ce730527]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 11: (ceph::osd::scheduler::PGOpItem::run(OSD*, OSDShard*, boost::intrusive_ptr<PG>&, ThreadPool::TPHandle&)+0x69) [0x55a0ce973de9]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 12: (OSD::ShardedOpWQ::_process(unsigned int, ceph::heartbeat_handle_d*)+0xab3) [0x55a0ce73b0d3]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 13: (ShardedThreadPool::shardedthreadpool_worker(unsigned int)+0x293) [0x55a0cec36b53]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 14: ceph-osd(+0xaa30b4) [0x55a0cec370b4]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 15: /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7f1cc9a42b43]
2024-07-19T02:41:53.787 INFO:tasks.ceph.osd.0.smithi037.stderr: 16: /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7f1cc9ad4a00]
2024-07-19T02:41:53.788 INFO:tasks.ceph.osd.0.smithi037.stderr:*** Caught signal (Aborted) **
2024-07-19T02:41:53.788 INFO:tasks.ceph.osd.0.smithi037.stderr: in thread 7f1caaa98640 thread_name:tp_osd_tp
2024-07-19T02:41:53.788 INFO:tasks.ceph.osd.0.smithi037.stderr:2024-07-19T02:41:53.782+0000 7f1caaa98640 -1 ./src/osd/ECCommon.cc: In function 'static void ECCommon::ReadPipeline::get_min_want_to_read_shards(uint64_t, uint64_t, const ECUtil::stripe_info_t&, const std::vector<int>&, std::set<int>*)' thread 7f1caaa98640 time 2024-07-19T02:41:53.781585+0000
2024-07-19T02:41:53.788 INFO:tasks.ceph.osd.0.smithi037.stderr:./src/osd/ECCommon.cc: 336: FAILED ceph_assert(want_to_read->size() == sinfo.get_data_chunk_count())

The recently introduced get_min_want_to_read_shards() draws assumptions on the contiguity of shard indexes in the want_to_read in/out parameter. For first call it's fine but when a read request is consisted of multiple regions (rarely), want_to_read is carried between multiple invocations (one call per region) making possible to violate the assert (the discussion: https://ceph-storage.slack.com/archives/C1HFZTW81/p1721387220588479).

The workaround is to disable EC partial reads (osd_ec_partial_reads = false).

Actions
Copy link
 #1

Updated by Radoslaw Zarzynski over 1 year ago ยท Edited

Replication

MON=1 OSD=3 MGR=1 MDS=0 RGW=0 ../src/vstart.sh -l -n --without-dashboard -o "rbd_default_data_pool=datapool" 

bin/ceph osd erasure-code-profile set teuthologyprofile crush-failure-domain=osd  m=1 k=2 &&
bin/ceph osd pool create datapool 4 4 erasure teuthologyprofile &&
bin/ceph osd pool set datapool allow_ec_overwrites true &&
bin/rbd pool init datapool

RBD_FEATURES=61 bin/ceph_test_librbd --gtest_filter="TestLibRBD.TestEncryptionLUKS1"

The problem

  -11> 2024-07-22T23:38:13.138+0000 7f7fc4577700 10 osd.0 pg_epoch: 21 pg[1.2s0( v 21'23 (0'0,21'23] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,NONE,1]p0(0) r=0 lpr=14 crt=21'23 lcod 
21'22 mlcod 21'22 active+undersized+degraded mbc={} trimq=[2~1]] do_osd_op 1:60a3bc56:::rbd_data.2.10321519e9c1.0000000000000001:head [read 512~512,read 16896~512]
...
    -4> 2024-07-22T23:38:13.138+0000 7f7fc4577700 20 osd.0 pg_epoch: 21 pg[1.2s0( v 21'23 (0'0,21'23] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,NONE,1]p0(0) r=0 lpr=14 crt=21'23 lcod
21'22 mlcod 21'22 active+undersized+degraded mbc={} trimq=[2~1]] get_min_want_to_read_shards before: offset 0 length 4096 want_to_read                                                                            
    -3> 2024-07-22T23:38:13.138+0000 7f7fc4577700 20 osd.0 pg_epoch: 21 pg[1.2s0( v 21'23 (0'0,21'23] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,NONE,1]p0(0) r=0 lpr=14 crt=21'23 lcod
21'22 mlcod 21'22 active+undersized+degraded mbc={} trimq=[2~1]] get_min_want_to_read_shards after: offset 0 length 4096 want_to_read 0                                                                           
    -2> 2024-07-22T23:38:13.138+0000 7f7fc4577700 20 osd.0 pg_epoch: 21 pg[1.2s0( v 21'23 (0'0,21'23] local-lis/les=14/15 n=2 ec=14/14 lis/c=14/0 les/c/f=15/0/0 sis=14) [0,NONE,1]p0(0) r=0 lpr=14 crt=21'23 lcod
21'22 mlcod 21'22 active+undersized+degraded mbc={} trimq=[2~1]] get_min_want_to_read_shards before: offset 16384 length 4096 want_to_read 0                                                                      
    -1> 2024-07-22T23:38:13.143+0000 7f7fc4577700 -1 /home/rzarzynski/ceph1/src/osd/ECCommon.cc: In function 'static void ECCommon::ReadPipeline::get_min_want_to_read_shards(uint64_t, uint64_t, const ECUtil::str
ipe_info_t&, const std::vector<int>&, std::set<int>*)' thread 7f7fc4577700 time 2024-07-22T23:38:13.139271+0000                                                                                                   
/home/rzarzynski/ceph1/src/osd/ECCommon.cc: 336: FAILED ceph_assert(want_to_read->size() == sinfo.get_data_chunk_count())   

--- a/src/osd/ECCommon.cc
+++ b/src/osd/ECCommon.cc
@@ -343,9 +343,11 @@ void ECCommon::ReadPipeline::get_min_want_to_read_shards(
   const uint64_t length,
   set<int> *want_to_read)
 {
+  dout(20) << __func__ << " before: offset " << offset << " length " << length
+          << " want_to_read " << *want_to_read << dendl;
   get_min_want_to_read_shards(
     offset, length, sinfo, ec_impl->get_chunk_mapping(), want_to_read);
-  dout(20) << __func__ << ": offset " << offset << " length " << length
+  dout(20) << __func__ << " after: offset " << offset << " length " << length
           << " want_to_read " << *want_to_read << dendl;
 }
Actions
Copy link
 #2

Updated by Radoslaw Zarzynski over 1 year ago

  • Status changed from In Progress to Fix Under Review
  • Pull request ID set to 58749
Actions
Copy link
 #3

Updated by Radoslaw Zarzynski over 1 year ago

scrub note: approved, awaits QA.

Actions
Copy link
 #4

Updated by Radoslaw Zarzynski over 1 year ago

scrub note: fixed the nits, still awaits QA.

Actions
Copy link
 #5

Updated by Laura Flores over 1 year ago

Awaits QA and a response to Ilya's comments.

Actions
Copy link

Also available in: Atom PDF