Project

General

Profile

Actions

Bug #70707

closed

mds.a asok_command: dump stray {prefix=dump stray} asserts MDS

Added by Adam Kupczyk 12 months ago. Updated 2 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Correctness/Safety
Target version:
% Done:

0%

Source:
Q/A
Backport:
reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v20.0.0-1536-g0f86e32274
Released In:
v20.2.0~538
Upkeep Timestamp:
2025-11-01T01:00:03+00:00

Description

Discovered in:
https://pulpito.ceph.com/akupczyk-2025-03-26_06:29:14-rados-aclamk-testing-nauvoo-2025-03-25-1601-distro-default-smithi/8210449/

    -4> 2025-03-26T07:05:39.949+0000 7f5748da1640  1 mds.a asok_command: dump loads {prefix=dump loads} (starting...)
    -3> 2025-03-26T07:05:39.949+0000 7f5748da1640  1 mds.a asok_command: dump snaps {prefix=dump snaps} (starting...)
    -2> 2025-03-26T07:05:39.949+0000 7f5748da1640  1 mds.a asok_command: dump stray {prefix=dump stray} (starting...)
    -1> 2025-03-26T07:05:39.953+0000 7f5748da1640 -1 ./src/common/Formatter.cc: In function 'virtual void ceph::JSONFormatter::close_section()' thread 7f5748da1640 time 2025-03-26T07:05:39.955480+0000
./src/common/Formatter.cc: 277: FAILED ceph_assert(!m_stack.empty())

 ceph version 20.0.0-754-gb910a5cc (b910a5cc35f98a492c38ce539136f262f57cc549) tentacle (dev)
 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x117) [0x7f574cedf6a3]
 2: /usr/lib/ceph/libceph-common.so.2(+0x186858) [0x7f574cedf858]
 3: /usr/lib/ceph/libceph-common.so.2(+0x17ee46) [0x7f574ced7e46]
 4: (MDCache::scan_stray_dir(dirfrag_t, std::unique_ptr<MDCache::C_MDS_DumpStrayDirCtx, std::default_delete<MDCache::C_MDS_DumpStrayDirCtx> >)+0xca) [0x5603a0b176ea]
 5: (MDCache::stray_status(std::unique_ptr<MDCache::C_MDS_DumpStrayDirCtx, std::default_delete<MDCache::C_MDS_DumpStrayDirCtx> >)+0x36) [0x5603a0b17da6]
 6: (MDSRankDispatcher::handle_asok_command(std::basic_string_view<char, std::char_traits<char> >, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > >, std::less<void>, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, boost::variant<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, bool, long, double, std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > > >, std::vector<long, std::allocator<long> >, std::vector<double, std::allocator<double> > > > > > const&, ceph::Formatter*, ceph::buffer::v15_2_0::list const&, std::function<void (int, std::basic_string_view<char, std::char_traits<char> >, ceph::buffer::v15_2_0::list&)>)+0x3408) [0x5603a09a8068]

Most likely caused by:
https://github.com/ceph/ceph/pull/60684


Related issues 3 (0 open3 closed)

Related to CephFS - Feature #56442: mds: build asok command to dump stray files and associated capsResolvedIgor Golikov

Actions
Copied to CephFS - Backport #71008: squid: mds.a asok_command: dump stray {prefix=dump stray} asserts MDSDuplicateIgor GolikovActions
Copied to CephFS - Backport #71009: reef: mds.a asok_command: dump stray {prefix=dump stray} asserts MDSDuplicateIgor GolikovActions
Actions #1

Updated by Laura Flores 12 months ago · Edited

/a/yuriw-2025-03-27_15:03:25-rados-wip-yuri7-testing-2025-03-26-1605-distro-default-smithi/8213289
/a/yuriw-2025-03-27_15:03:25-rados-wip-yuri7-testing-2025-03-26-1605-distro-default-smithi/8213424

Actions #2

Updated by Laura Flores 12 months ago

  • Priority changed from Normal to High
Actions #3

Updated by Igor Golikov 12 months ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 62578
Actions #4

Updated by Sridhar Seshasayee 12 months ago

/a/skanta-2025-03-27_08:02:07-rados-wip-bharath10-testing-2025-03-27-0430-distro-default-smithi/8212813

Actions #5

Updated by Aishwarya Mathuria 12 months ago

  • Tags set to main-failures

/a//teuthology-2025-04-06_20:00:14-rados-main-distro-default-smithi/8226460

Actions #6

Updated by Laura Flores 12 months ago

/a/skanta-2025-04-04_06:10:17-rados-wip-bharath10-testing-2025-04-03-2112-distro-default-smithi/8223710

Actions #7

Updated by Igor Golikov 12 months ago

There is open PR for this fix
https://github.com/ceph/ceph/pull/62578
Does the last runs with the failure include this fix?

Actions #8

Updated by Sridhar Seshasayee 12 months ago

/a/yuriw-2025-04-07_23:32:09-rados-wip-yuri13-testing-2025-04-07-1144-distro-default-smithi/8229275

The fix is yet to be merged in main.

Actions #9

Updated by Laura Flores 12 months ago · Edited

/a/teuthology-2025-04-06_20:00:14-rados-main-distro-default-smithi/8226460
/a/teuthology-2025-04-06_20:00:14-rados-main-distro-default-smithi/8226612

Actions #10

Updated by Kamoltat (Junior) Sirivadhna 11 months ago

/a/skanta-2025-04-09_05:31:19-rados-wip-bharath17-testing-2025-04-08-0602-distro-default-smithi/
[8233192, 8233201]

Actions #11

Updated by Laura Flores 11 months ago

/a/lflores-2025-04-11_15:58:32-rados-wip-lflores-testing-2025-04-10-2245-distro-default-smithi/8235746

Actions #12

Updated by Laura Flores 11 months ago

/a/lflores-2025-04-15_00:57:49-rados-wip-lflores-testing-5-2025-04-14-1635-distro-default-gibba/8241448

Actions #13

Updated by Kamoltat (Junior) Sirivadhna 11 months ago

/a/lflores-2025-04-14_14:00:23-rados-wip-lflores-testing-2-2025-04-11-1133-distro-default-smithi/8239412

Actions #14

Updated by Laura Flores 11 months ago

/a/lflores-2025-04-11_19:10:45-rados-wip-lflores-testing-3-2025-04-11-1140-distro-default-smithi/8236029

Actions #15

Updated by Kamoltat (Junior) Sirivadhna 11 months ago

lflores-2025-04-14_22:30:34-rados-wip-lflores-testing-2025-04-10-2245-distro-default-gibba/

8241116
8241124

Actions #16

Updated by Venky Shankar 11 months ago

  • Category set to Correctness/Safety
  • Status changed from Fix Under Review to Pending Backport
  • Target version set to v20.0.0
  • Source set to Q/A
  • Backport set to reef,squid
Actions #17

Updated by Venky Shankar 11 months ago

  • Related to Feature #56442: mds: build asok command to dump stray files and associated caps added
Actions #18

Updated by Venky Shankar 11 months ago

@Igor Golikov - please pull in this fix with backports for https://tracker.ceph.com/issues/56442

Actions #20

Updated by Upkeep Bot 11 months ago

  • Copied to Backport #71008: squid: mds.a asok_command: dump stray {prefix=dump stray} asserts MDS added
Actions #21

Updated by Upkeep Bot 11 months ago

  • Copied to Backport #71009: reef: mds.a asok_command: dump stray {prefix=dump stray} asserts MDS added
Actions #22

Updated by Upkeep Bot 11 months ago

  • Tags (freeform) set to backport_processed
Actions #23

Updated by Sridhar Seshasayee 11 months ago

/a/skanta-2025-04-22_23:21:15-rados-wip-bharath1-testing-2025-04-21-0529-distro-default-smithi/
[8254488, 8254505]

Actions #24

Updated by Aishwarya Mathuria 11 months ago

/a/yuriw-2025-04-14_18:07:07-rados-wip-yuri10-testing-2025-04-08-0710-distro-default-smithi/8239955
/a/yuriw-2025-04-14_18:07:07-rados-wip-yuri10-testing-2025-04-08-0710-distro-default-smithi/8239968

Actions #25

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 0f86e32274fe7f608989c8ceab536a415e5b79b5
  • Fixed In set to v20.0.0-1536-g0f86e32274f
  • Upkeep Timestamp set to 2025-07-08T14:47:05+00:00
Actions #26

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v20.0.0-1536-g0f86e32274f to v20.0.0-1536-g0f86e32274fe
  • Upkeep Timestamp changed from 2025-07-08T14:47:05+00:00 to 2025-07-14T15:21:12+00:00
Actions #27

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v20.0.0-1536-g0f86e32274fe to v20.0.0-1536-g0f86e32274
  • Upkeep Timestamp changed from 2025-07-14T15:21:12+00:00 to 2025-07-14T20:45:45+00:00
Actions #28

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~538
  • Upkeep Timestamp changed from 2025-07-14T20:45:45+00:00 to 2025-11-01T01:00:03+00:00
Actions #29

Updated by Igor Golikov 2 months ago

  • Status changed from Pending Backport to Resolved
Actions

Also available in: Atom PDF