Bug #71547: BlueFS.mkfs segfaults - bluestore - Ceph

Actions

Copy link

Bug #71547

closed

BlueFS.mkfs segfaults

Added by Kefu Chai 10 months ago. Updated 7 months ago.

Status:

Resolved

Priority:

Normal

Assignee:

Kefu Chai

Target version:

% Done:

Source:

Backport:

Regression:

Severity:

3 - minor

Reviewed:

Affected Versions:

ceph-qa-suite:

Pull request ID:

64273

Tags (freeform):

Merge Commit:

da1b72e4874fe3f786a7011176b97aa2f12c8f83

Fixed In:

v20.3.0-2699-gda1b72e487

Released In:

Upkeep Timestamp:

2025-09-01T23:45:23+00:00

Tags:

arm64

Description

we had 3 failures in a row, all of them segfaulted at the same place:

objectstore.unittest_bluefs¶

[ RUN      ] BlueFS.mkfs
*** Caught signal (Segmentation fault) **
 in thread 7fb868972600 thread_name:unittest_bluefs
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0224) [0x562167446224]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb86954f520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x5621672f557b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x5621672f53f5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x5621672f5315]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x562167541625]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56216744e584]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56216737accc]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5621672a86db]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x56216735d2d4]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x562167340896]
 12: (testing::Test::Run()+0xb7) [0x562167321737]
 13: (testing::TestInfo::Run()+0x10f) [0x56216732214f]
 14: (testing::TestSuite::Run()+0x17a) [0x56216732296a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5621673327c8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x562167361414]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x562167342d76]
 18: (testing::UnitTest::Run()+0xb6) [0x562167332356]
 19: (RUN_ALL_TESTS()+0x11) [0x5621672f0901]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb869536d90]
 22: __libc_start_main()
 23: _start()
2025-06-03T21:01:55.636+0000 7fb868972600 -1 WARNING: all dangerous and experimental features are enabled.
2025-06-03T21:01:55.636+0000 7fb868972600  1 bdev(0x562168f2cf00 ceph_test_bluefs.tmp.block.1314733.1) open path ceph_test_bluefs.tmp.block.1314733.1
2025-06-03T21:01:55.640+0000 7fb868972600 -1 *** Caught signal (Segmentation fault) **
 in thread 7fb868972600 thread_name:unittest_bluefs

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0224) [0x562167446224]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb86954f520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x5621672f557b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x5621672f53f5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x5621672f5315]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x562167541625]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56216744e584]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56216737accc]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5621672a86db]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x56216735d2d4]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x562167340896]
 12: (testing::Test::Run()+0xb7) [0x562167321737]
 13: (testing::TestInfo::Run()+0x10f) [0x56216732214f]
 14: (testing::TestSuite::Run()+0x17a) [0x56216732296a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5621673327c8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x562167361414]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x562167342d76]
 18: (testing::UnitTest::Run()+0xb6) [0x562167332356]
 19: (RUN_ALL_TESTS()+0x11) [0x5621672f0901]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb869536d90]
 22: __libc_start_main()
 23: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -35> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command assert hook 0x562168d54ad0
   -34> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command abort hook 0x562168d54ad0
   -33> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command leak_some_memory hook 0x562168d54ad0
   -32> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command perfcounters_dump hook 0x562168d54ad0
   -31> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command 1 hook 0x562168d54ad0
   -30> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command perf dump hook 0x562168d54ad0
   -29> 2025-06-03T21:01:55.612+0000 7fb868972600  5 asok(0x562168e3c000) register_command perfcounters_schema hook 0x562168d54ad0
   -28> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command perf histogram dump hook 0x562168d54ad0
   -27> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command 2 hook 0x562168d54ad0
   -26> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command perf schema hook 0x562168d54ad0
   -25> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command counter dump hook 0x562168d54ad0
   -24> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command counter schema hook 0x562168d54ad0
   -23> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command perf histogram schema hook 0x562168d54ad0
   -22> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command perf reset hook 0x562168d54ad0
   -21> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config show hook 0x562168d54ad0
   -20> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config help hook 0x562168d54ad0
   -19> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config set hook 0x562168d54ad0
   -18> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config unset hook 0x562168d54ad0
   -17> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config get hook 0x562168d54ad0
   -16> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config diff hook 0x562168d54ad0
   -15> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command config diff get hook 0x562168d54ad0
   -14> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command injectargs hook 0x562168d54ad0
   -13> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command log flush hook 0x562168d54ad0
   -12> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command log dump hook 0x562168d54ad0
   -11> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command log reopen hook 0x562168d54ad0
   -10> 2025-06-03T21:01:55.616+0000 7fb868972600  5 asok(0x562168e3c000) register_command dump_mempools hook 0x562168eaec08
    -9> 2025-06-03T21:01:55.636+0000 7fb868972600 -1 WARNING: all dangerous and experimental features are enabled.
    -8> 2025-06-03T21:01:55.636+0000 7fb868972600  5 asok(0x562168e3c000) register_command bluestore bluefs device info hook 0x562168d54ab0
    -7> 2025-06-03T21:01:55.636+0000 7fb868972600  5 asok(0x562168e3c000) register_command bluefs stats hook 0x562168d54ab0
    -6> 2025-06-03T21:01:55.636+0000 7fb868972600  5 asok(0x562168e3c000) register_command bluefs files list hook 0x562168d54ab0
    -5> 2025-06-03T21:01:55.636+0000 7fb868972600  5 asok(0x562168e3c000) register_command bluefs debug_inject_read_zeros hook 0x562168d54ab0
    -4> 2025-06-03T21:01:55.636+0000 7fb868972600 10 bluefs add_block_device bdev 1 path ceph_test_bluefs.tmp.block.1314733.1 
    -3> 2025-06-03T21:01:55.636+0000 7fb868972600  1 bdev(0x562168f2cf00 ceph_test_bluefs.tmp.block.1314733.1) open path ceph_test_bluefs.tmp.block.1314733.1
    -2> 2025-06-03T21:01:55.636+0000 7fb868972600 10 bdev(0x562168f2cf00 ceph_test_bluefs.tmp.block.1314733.1) _lock fd=6
    -1> 2025-06-03T21:01:55.636+0000 7fb868972600 20 bdev(0x562168f2cf00 ceph_test_bluefs.tmp.block.1314733.1) open devname sdc
     0> 2025-06-03T21:01:55.640+0000 7fb868972600 -1 *** Caught signal (Segmentation fault) **
 in thread 7fb868972600 thread_name:unittest_bluefs

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0224) [0x562167446224]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fb86954f520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x5621672f557b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x5621672f53f5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x5621672f5315]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x562167541625]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56216744e584]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56216737accc]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5621672a86db]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x56216735d2d4]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x562167340896]
 12: (testing::Test::Run()+0xb7) [0x562167321737]
 13: (testing::TestInfo::Run()+0x10f) [0x56216732214f]
 14: (testing::TestSuite::Run()+0x17a) [0x56216732296a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5621673327c8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x562167361414]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x562167342d76]
 18: (testing::UnitTest::Run()+0xb6) [0x562167332356]
 19: (RUN_ALL_TESTS()+0x11) [0x5621672f0901]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fb869536d90]
 22: __libc_start_main()
 23: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

objectstore.unittest_bdev¶

[ RUN      ] KernelDevice.Ticket45337
2025-06-03T21:01:57.416+0000 7f69c2ea3600 -1 WARNING: all dangerous and experimental features are enabled.
*** Caught signal (Segmentation fault) **
 in thread 7f69c2ea3600 thread_name:unittest_bdev
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1bba54) [0x560ad8c54a54]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f69c3a80520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x560ad8bd481b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x560ad8bd4695]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x560ad8bd45b5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x560ad8d56cc5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x560ad8c5e124]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x560ad8bcf447]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x560ad8c2f624]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x560ad8c11e66]
 11: (testing::Test::Run()+0xb7) [0x560ad8bf23b7]
 12: (testing::TestInfo::Run()+0x10f) [0x560ad8bf2dcf]
 13: (testing::TestSuite::Run()+0x17a) [0x560ad8bf35ea]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x560ad8c03448]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x560ad8c33764]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x560ad8c143c6]
 17: (testing::UnitTest::Run()+0xb6) [0x560ad8c02fd6]
 18: (RUN_ALL_TESTS()+0x11) [0x560ad8bd14c1]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f69c3a67d90]
 21: __libc_start_main()
 22: _start()
2025-06-03T21:02:06.116+0000 7f69c2ea3600  1 bdev(0x560adab38f00 ceph_test_bluefs.tmp.block.1314981.1) open path ceph_test_bluefs.tmp.block.1314981.1
2025-06-03T21:02:06.120+0000 7f69c2ea3600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f69c2ea3600 thread_name:unittest_bdev

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1bba54) [0x560ad8c54a54]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f69c3a80520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x560ad8bd481b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x560ad8bd4695]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x560ad8bd45b5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x560ad8d56cc5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x560ad8c5e124]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x560ad8bcf447]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x560ad8c2f624]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x560ad8c11e66]
 11: (testing::Test::Run()+0xb7) [0x560ad8bf23b7]
 12: (testing::TestInfo::Run()+0x10f) [0x560ad8bf2dcf]
 13: (testing::TestSuite::Run()+0x17a) [0x560ad8bf35ea]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x560ad8c03448]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x560ad8c33764]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x560ad8c143c6]
 17: (testing::UnitTest::Run()+0xb6) [0x560ad8c02fd6]
 18: (RUN_ALL_TESTS()+0x11) [0x560ad8bd14c1]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f69c3a67d90]
 21: __libc_start_main()
 22: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -30> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command assert hook 0x560ada960ae0
   -29> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command abort hook 0x560ada960ae0
   -28> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command leak_some_memory hook 0x560ada960ae0
   -27> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perfcounters_dump hook 0x560ada960ae0
   -26> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command 1 hook 0x560ada960ae0
   -25> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perf dump hook 0x560ada960ae0
   -24> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perfcounters_schema hook 0x560ada960ae0
   -23> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perf histogram dump hook 0x560ada960ae0
   -22> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command 2 hook 0x560ada960ae0
   -21> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perf schema hook 0x560ada960ae0
   -20> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command counter dump hook 0x560ada960ae0
   -19> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command counter schema hook 0x560ada960ae0
   -18> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perf histogram schema hook 0x560ada960ae0
   -17> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command perf reset hook 0x560ada960ae0
   -16> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config show hook 0x560ada960ae0
   -15> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config help hook 0x560ada960ae0
   -14> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config set hook 0x560ada960ae0
   -13> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config unset hook 0x560ada960ae0
   -12> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config get hook 0x560ada960ae0
   -11> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config diff hook 0x560ada960ae0
   -10> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command config diff get hook 0x560ada960ae0
    -9> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command injectargs hook 0x560ada960ae0
    -8> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command log flush hook 0x560ada960ae0
    -7> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command log dump hook 0x560ada960ae0
    -6> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command log reopen hook 0x560ada960ae0
    -5> 2025-06-03T21:01:57.408+0000 7f69c2ea3600  5 asok(0x560adaa44000) register_command dump_mempools hook 0x560adab40c08
    -4> 2025-06-03T21:01:57.416+0000 7f69c2ea3600 -1 WARNING: all dangerous and experimental features are enabled.
    -3> 2025-06-03T21:02:06.116+0000 7f69c2ea3600  1 bdev(0x560adab38f00 ceph_test_bluefs.tmp.block.1314981.1) open path ceph_test_bluefs.tmp.block.1314981.1
    -2> 2025-06-03T21:02:06.120+0000 7f69c2ea3600 10 bdev(0x560adab38f00 ceph_test_bluefs.tmp.block.1314981.1) _lock fd=6
    -1> 2025-06-03T21:02:06.120+0000 7f69c2ea3600 20 bdev(0x560adab38f00 ceph_test_bluefs.tmp.block.1314981.1) open devname sdc
     0> 2025-06-03T21:02:06.120+0000 7f69c2ea3600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f69c2ea3600 thread_name:unittest_bdev

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1bba54) [0x560ad8c54a54]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f69c3a80520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xcb) [0x560ad8bd481b]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x560ad8bd4695]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x560ad8bd45b5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x560ad8d56cc5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x560ad8c5e124]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x560ad8bcf447]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x560ad8c2f624]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x560ad8c11e66]
 11: (testing::Test::Run()+0xb7) [0x560ad8bf23b7]
 12: (testing::TestInfo::Run()+0x10f) [0x560ad8bf2dcf]
 13: (testing::TestSuite::Run()+0x17a) [0x560ad8bf35ea]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x560ad8c03448]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x560ad8c33764]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x560ad8c143c6]
 17: (testing::UnitTest::Run()+0xb6) [0x560ad8c02fd6]
 18: (RUN_ALL_TESTS()+0x11) [0x560ad8bd14c1]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f69c3a67d90]
 21: __libc_start_main()
 22: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

objectstore.unittest_bluefs_ex¶

[ RUN      ] BlueFS_ex.test_interrupted_compaction
waiting for compaction to terminate
growing BlueFS log for async compact, stop at #1
*** Caught signal (Segmentation fault) **
 in thread 7fc43bc2a600 thread_name:unittest_bluefs
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs_ex(+0x3531c4) [0x55cbed7ec1c4]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fc43c807520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x55cbed6a4d17]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x55cbed6a4b95]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x55cbed6a4ab5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x55cbed8e79b5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x55cbed7f4524]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x55cbed708e0c]
 9: (BlueFS_ex::grow_log_interrupt_on_compact(int, unsigned int)+0x13a) [0x55cbed69cc2a]
 10: (BlueFS_ex_test_interrupted_compaction_Test::TestBody()+0x3e1) [0x55cbed69b711]
 11: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x55cbed6ff144]
 12: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x55cbed6e1b26]
 13: (testing::Test::Run()+0xb7) [0x55cbed6c2297]
 14: (testing::TestInfo::Run()+0x10f) [0x55cbed6c2caf]
 15: (testing::TestSuite::Run()+0x17a) [0x55cbed6c34ca]
 16: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x55cbed6d3328]
 17: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x55cbed703284]
 18: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x55cbed6e4086]
 19: (testing::UnitTest::Run()+0xb6) [0x55cbed6d2eb6]
 20: (RUN_ALL_TESTS()+0x11) [0x55cbed69df61]
 21: main()
 22: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7fc43c7eed90]
 23: __libc_start_main()
 24: _start()
done code=0
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs_ex.cc:180: Failure
Value of: (((stat) & 0xff00) >> 8) == 0
  Actual: false
Expected: true

these failures surfaced when testing a packaging related pull request at https://github.com/ceph/ceph/pull/63702

we had a similar issue reported at #69909

Files

consoleText.71547.gz (184 KB) consoleText.71547.gz

Kefu Chai, 06/04/2025 03:30 AM

Related issues 1 (0 open — 1 closed)

Actions

Copy link

Updated by Kefu Chai 10 months ago

Copied from Bug #69909: BlueFS.mkfs fails from Bus error on arm64 added

Actions

Copy link

Updated by Kefu Chai 10 months ago

File deleted (~~#68049.txt.gz~~)

Actions

Copy link

Updated by Kefu Chai 10 months ago

File consoleText.71547.gz consoleText.71547.gz added

Actions

Copy link

Updated by Kefu Chai 9 months ago

Status changed from New to Fix Under Review
Assignee set to Kefu Chai
Pull request ID set to 64273

Actions

Copy link

Updated by Jonathan Bailey 7 months ago · Edited

I believe I've seen this same issue on x86_64 (It was in Jenkins make check - https://jenkins.ceph.com/job/ceph-pull-requests/165123/consoleFull ). Stack traces look to be the same from a Quick Look, which makes me think this is not isolated to arm64:

unittest_bluefs

[ RUN      ] BlueFS.mkfs
2025-08-18T08:32:17.159+0000 7f37e1920600  1 bdev(0x563962d88f00 ceph_test_bluefs.tmp.block.463643.1) open path ceph_test_bluefs.tmp.block.463643.1
*** Caught signal (Segmentation fault) **
 in thread 7f37e1920600 thread_name:unittest_bluefs
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0da4) [0x563930655da4]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f37e24fd520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x5639305049e7]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x563930504865]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x563930504785]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x5639307542e5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56393065e514]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56393058b79c]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5639304b725b]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x563930571b54]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x563930550506]
 12: (testing::Test::Run()+0xb7) [0x563930531257]
 13: (testing::TestInfo::Run()+0x10f) [0x563930531c6f]
 14: (testing::TestSuite::Run()+0x17a) [0x56393053248a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5639305422e8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x563930573324]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x5639305529e6]
 18: (testing::UnitTest::Run()+0xb6) [0x563930541e76]
 19: (RUN_ALL_TESTS()+0x11) [0x5639304ff8e1]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f37e24e4d90]
 22: __libc_start_main()
 23: _start()
2025-08-18T08:32:17.171+0000 7f37e1920600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f37e1920600 thread_name:unittest_bluefs

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0da4) [0x563930655da4]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f37e24fd520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x5639305049e7]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x563930504865]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x563930504785]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x5639307542e5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56393065e514]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56393058b79c]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5639304b725b]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x563930571b54]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x563930550506]
 12: (testing::Test::Run()+0xb7) [0x563930531257]
 13: (testing::TestInfo::Run()+0x10f) [0x563930531c6f]
 14: (testing::TestSuite::Run()+0x17a) [0x56393053248a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5639305422e8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x563930573324]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x5639305529e6]
 18: (testing::UnitTest::Run()+0xb6) [0x563930541e76]
 19: (RUN_ALL_TESTS()+0x11) [0x5639304ff8e1]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f37e24e4d90]
 22: __libc_start_main()
 23: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -35> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command assert hook 0x563962bc8a60
   -34> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command abort hook 0x563962bc8a60
   -33> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command leak_some_memory hook 0x563962bc8a60
   -32> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perfcounters_dump hook 0x563962bc8a60
   -31> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command 1 hook 0x563962bc8a60
   -30> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perf dump hook 0x563962bc8a60
   -29> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perfcounters_schema hook 0x563962bc8a60
   -28> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perf histogram dump hook 0x563962bc8a60
   -27> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command 2 hook 0x563962bc8a60
   -26> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perf schema hook 0x563962bc8a60
   -25> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command counter dump hook 0x563962bc8a60
   -24> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command counter schema hook 0x563962bc8a60
   -23> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perf histogram schema hook 0x563962bc8a60
   -22> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command perf reset hook 0x563962bc8a60
   -21> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config show hook 0x563962bc8a60
   -20> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config help hook 0x563962bc8a60
   -19> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config set hook 0x563962bc8a60
   -18> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config unset hook 0x563962bc8a60
   -17> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config get hook 0x563962bc8a60
   -16> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config diff hook 0x563962bc8a60
   -15> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command config diff get hook 0x563962bc8a60
   -14> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command injectargs hook 0x563962bc8a60
   -13> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command log flush hook 0x563962bc8a60
   -12> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command log dump hook 0x563962bc8a60
   -11> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command log reopen hook 0x563962bc8a60
   -10> 2025-08-18T08:32:17.127+0000 7f37e1920600  5 asok(0x563962cb2000) register_command dump_mempools hook 0x563962d24c08
    -9> 2025-08-18T08:32:17.147+0000 7f37e1920600 -1 WARNING: all dangerous and experimental features are enabled.
    -8> 2025-08-18T08:32:17.151+0000 7f37e1920600  5 asok(0x563962cb2000) register_command bluestore bluefs device info hook 0x563962bc8a40
    -7> 2025-08-18T08:32:17.151+0000 7f37e1920600  5 asok(0x563962cb2000) register_command bluefs stats hook 0x563962bc8a40
    -6> 2025-08-18T08:32:17.151+0000 7f37e1920600  5 asok(0x563962cb2000) register_command bluefs files list hook 0x563962bc8a40
    -5> 2025-08-18T08:32:17.151+0000 7f37e1920600  5 asok(0x563962cb2000) register_command bluefs debug_inject_read_zeros hook 0x563962bc8a40
    -4> 2025-08-18T08:32:17.159+0000 7f37e1920600 10 bluefs add_block_device bdev 1 path ceph_test_bluefs.tmp.block.463643.1 
    -3> 2025-08-18T08:32:17.159+0000 7f37e1920600  1 bdev(0x563962d88f00 ceph_test_bluefs.tmp.block.463643.1) open path ceph_test_bluefs.tmp.block.463643.1
    -2> 2025-08-18T08:32:17.159+0000 7f37e1920600 10 bdev(0x563962d88f00 ceph_test_bluefs.tmp.block.463643.1) _lock fd=6
    -1> 2025-08-18T08:32:17.159+0000 7f37e1920600 20 bdev(0x563962d88f00 ceph_test_bluefs.tmp.block.463643.1) open devname sdc
     0> 2025-08-18T08:32:17.171+0000 7f37e1920600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f37e1920600 thread_name:unittest_bluefs

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs(+0x3c0da4) [0x563930655da4]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f37e24fd520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x5639305049e7]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x563930504865]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x563930504785]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x5639307542e5]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x56393065e514]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x56393058b79c]
 9: (BlueFS_mkfs_Test::TestBody()+0x6b) [0x5639304b725b]
 10: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x563930571b54]
 11: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x563930550506]
 12: (testing::Test::Run()+0xb7) [0x563930531257]
 13: (testing::TestInfo::Run()+0x10f) [0x563930531c6f]
 14: (testing::TestSuite::Run()+0x17a) [0x56393053248a]
 15: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x5639305422e8]
 16: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x563930573324]
 17: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x5639305529e6]
 18: (testing::UnitTest::Run()+0xb6) [0x563930541e76]
 19: (RUN_ALL_TESTS()+0x11) [0x5639304ff8e1]
 20: main()
 21: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f37e24e4d90]
 22: __libc_start_main()
 23: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

BlueFS_ex

197/318 Test #210: unittest_bluefs_ex ........................***Failed    0.74 sec
[==========] Running 1 test from 1 test suite.
[----------] Global test environment set-up.
[----------] 1 test from BlueFS_ex
[ RUN      ] BlueFS_ex.test_interrupted_compaction
waiting for compaction to terminate
growing BlueFS log for async compact, stop at #1
*** Caught signal (Segmentation fault) **
 in thread 7f363722e600 thread_name:unittest_bluefs
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bluefs_ex(+0x3538e4) [0x560f2ebe78e4]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f3637e0b520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x560f2ea9fc77]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x560f2ea9faf5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x560f2ea9fa15]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x560f2ece6305]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x560f2ebf0054]
 8: (BlueFS::add_block_device(unsigned int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bluefs_shared_alloc_context_t*)+0x38c) [0x560f2eb04c3c]
 9: (BlueFS_ex::grow_log_interrupt_on_compact(int, unsigned int)+0x13a) [0x560f2ea9779a]
 10: (BlueFS_ex_test_interrupted_compaction_Test::TestBody()+0x3e1) [0x560f2ea96291]
 11: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x560f2eaff2c4]
 12: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x560f2eadd0a6]
 13: (testing::Test::Run()+0xb7) [0x560f2eabd6d7]
 14: (testing::TestInfo::Run()+0x10f) [0x560f2eabe0ef]
 15: (testing::TestSuite::Run()+0x17a) [0x560f2eabe90a]
 16: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x560f2eace768]
 17: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x560f2eb00a94]
 18: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x560f2eadf606]
 19: (testing::UnitTest::Run()+0xb6) [0x560f2eace2f6]
 20: (RUN_ALL_TESTS()+0x11) [0x560f2ea98ad1]
 21: main()
 22: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f3637df2d90]
 23: __libc_start_main()
 24: _start()
done code=0
/home/jenkins-build/build/workspace/ceph-pull-requests/src/test/objectstore/test_bluefs_ex.cc:180: Failure
Value of: (((stat) & 0xff00) >> 8) == 0
  Actual: false
Expected: true

unittest_bdev

[ RUN      ] KernelDevice.Ticket45337
2025-08-18T08:32:29.043+0000 7f7a47fce600 -1 WARNING: all dangerous and experimental features are enabled.
2025-08-18T08:32:39.194+0000 7f7a47fce600  1 bdev(0x55f7c99d0f00 ceph_test_bluefs.tmp.block.465621.1) open path ceph_test_bluefs.tmp.block.465621.1
*** Caught signal (Segmentation fault) **
 in thread 7f7a47fce600 thread_name:unittest_bdev
 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1ba204) [0x55f79ca86204]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7a48bab520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x55f79ca06857]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x55f79ca066d5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x55f79ca065f5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x55f79cb8ab45]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x55f79ca8f5c4]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x55f79ca000c7]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x55f79ca658a4]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x55f79ca434e6]
 11: (testing::Test::Run()+0xb7) [0x55f79ca238f7]
 12: (testing::TestInfo::Run()+0x10f) [0x55f79ca2430f]
 13: (testing::TestSuite::Run()+0x17a) [0x55f79ca24b2a]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x55f79ca34988]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x55f79ca67074]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x55f79ca45a46]
 17: (testing::UnitTest::Run()+0xb6) [0x55f79ca34516]
 18: (RUN_ALL_TESTS()+0x11) [0x55f79ca02111]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f7a48b92d90]
 21: __libc_start_main()
 22: _start()
2025-08-18T08:32:39.198+0000 7f7a47fce600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f7a47fce600 thread_name:unittest_bdev

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1ba204) [0x55f79ca86204]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7a48bab520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x55f79ca06857]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x55f79ca066d5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x55f79ca065f5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x55f79cb8ab45]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x55f79ca8f5c4]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x55f79ca000c7]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x55f79ca658a4]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x55f79ca434e6]
 11: (testing::Test::Run()+0xb7) [0x55f79ca238f7]
 12: (testing::TestInfo::Run()+0x10f) [0x55f79ca2430f]
 13: (testing::TestSuite::Run()+0x17a) [0x55f79ca24b2a]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x55f79ca34988]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x55f79ca67074]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x55f79ca45a46]
 17: (testing::UnitTest::Run()+0xb6) [0x55f79ca34516]
 18: (RUN_ALL_TESTS()+0x11) [0x55f79ca02111]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f7a48b92d90]
 21: __libc_start_main()
 22: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

--- begin dump of recent events ---
   -30> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command assert hook 0x55f7c9810a70
   -29> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command abort hook 0x55f7c9810a70
   -28> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command leak_some_memory hook 0x55f7c9810a70
   -27> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perfcounters_dump hook 0x55f7c9810a70
   -26> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command 1 hook 0x55f7c9810a70
   -25> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perf dump hook 0x55f7c9810a70
   -24> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perfcounters_schema hook 0x55f7c9810a70
   -23> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perf histogram dump hook 0x55f7c9810a70
   -22> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command 2 hook 0x55f7c9810a70
   -21> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perf schema hook 0x55f7c9810a70
   -20> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command counter dump hook 0x55f7c9810a70
   -19> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command counter schema hook 0x55f7c9810a70
   -18> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perf histogram schema hook 0x55f7c9810a70
   -17> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command perf reset hook 0x55f7c9810a70
   -16> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config show hook 0x55f7c9810a70
   -15> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config help hook 0x55f7c9810a70
   -14> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config set hook 0x55f7c9810a70
   -13> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config unset hook 0x55f7c9810a70
   -12> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config get hook 0x55f7c9810a70
   -11> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config diff hook 0x55f7c9810a70
   -10> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command config diff get hook 0x55f7c9810a70
    -9> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command injectargs hook 0x55f7c9810a70
    -8> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command log flush hook 0x55f7c9810a70
    -7> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command log dump hook 0x55f7c9810a70
    -6> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command log reopen hook 0x55f7c9810a70
    -5> 2025-08-18T08:32:29.035+0000 7f7a47fce600  5 asok(0x55f7c98f4000) register_command dump_mempools hook 0x55f7c98f0c08
    -4> 2025-08-18T08:32:29.043+0000 7f7a47fce600 -1 WARNING: all dangerous and experimental features are enabled.
    -3> 2025-08-18T08:32:39.194+0000 7f7a47fce600  1 bdev(0x55f7c99d0f00 ceph_test_bluefs.tmp.block.465621.1) open path ceph_test_bluefs.tmp.block.465621.1
    -2> 2025-08-18T08:32:39.194+0000 7f7a47fce600 10 bdev(0x55f7c99d0f00 ceph_test_bluefs.tmp.block.465621.1) _lock fd=6
    -1> 2025-08-18T08:32:39.194+0000 7f7a47fce600 20 bdev(0x55f7c99d0f00 ceph_test_bluefs.tmp.block.465621.1) open devname sdc
     0> 2025-08-18T08:32:39.198+0000 7f7a47fce600 -1 *** Caught signal (Segmentation fault) **
 in thread 7f7a47fce600 thread_name:unittest_bdev

 ceph version Development (no_version) tentacle (dev - Debug)
 1: /home/jenkins-build/build/workspace/ceph-pull-requests/build/bin/unittest_bdev(+0x1ba204) [0x55f79ca86204]
 2: /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7f7a48bab520]
 3: (ceph::mutex_debug_detail::mutex_debugging_base::is_locked_by_me() const+0xc7) [0x55f79ca06857]
 4: (ceph::mutex_debug_detail::mutex_debug_impl<false>::lock(bool)+0x25) [0x55f79ca066d5]
 5: (std::lock_guard<ceph::mutex_debug_detail::mutex_debug_impl<false> >::lock_guard(ceph::mutex_debug_detail::mutex_debug_impl<false>&)+0x25) [0x55f79ca065f5]
 6: (ceph::extblkdev::detect_device(ceph::common::CephContext*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::shared_ptr<ceph::ExtBlkDevInterface>&)+0x55) [0x55f79cb8ab45]
 7: (KernelDevice::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)+0x1b34) [0x55f79ca8f5c4]
 8: (KernelDevice_Ticket45337_Test::TestBody()+0x2b7) [0x55f79ca000c7]
 9: (void testing::internal::HandleSehExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x74) [0x55f79ca658a4]
 10: (void testing::internal::HandleExceptionsInMethodIfSupported<testing::Test, void>(testing::Test*, void (testing::Test::*)(), char const*)+0x66) [0x55f79ca434e6]
 11: (testing::Test::Run()+0xb7) [0x55f79ca238f7]
 12: (testing::TestInfo::Run()+0x10f) [0x55f79ca2430f]
 13: (testing::TestSuite::Run()+0x17a) [0x55f79ca24b2a]
 14: (testing::internal::UnitTestImpl::RunAllTests()+0x418) [0x55f79ca34988]
 15: (bool testing::internal::HandleSehExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x74) [0x55f79ca67074]
 16: (bool testing::internal::HandleExceptionsInMethodIfSupported<testing::internal::UnitTestImpl, bool>(testing::internal::UnitTestImpl*, bool (testing::internal::UnitTestImpl::*)(), char const*)+0x66) [0x55f79ca45a46]
 17: (testing::UnitTest::Run()+0xb6) [0x55f79ca34516]
 18: (RUN_ALL_TESTS()+0x11) [0x55f79ca02111]
 19: main()
 20: /lib/x86_64-linux-gnu/libc.so.6(+0x29d90) [0x7f7a48b92d90]
 21: __libc_start_main()
 22: _start()
 NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.

Actions

Copy link

Updated by Upkeep Bot 7 months ago

Status changed from Fix Under Review to Resolved
Merge Commit set to da1b72e4874fe3f786a7011176b97aa2f12c8f83
Fixed In set to v20.3.0-2699-gda1b72e487
Upkeep Timestamp set to 2025-09-01T23:45:23+00:00

Actions

Copy link

Also available in: Atom PDF

Project

General

Profile

Ceph » bluestore

Tags

Custom queries

Bug #71547

BlueFS.mkfs segfaults

objectstore.unittest_bluefs¶

objectstore.unittest_bdev¶

objectstore.unittest_bluefs_ex¶

Updated by Kefu Chai 10 months ago

Updated by Kefu Chai 10 months ago

Updated by Kefu Chai 10 months ago

Updated by Kefu Chai 9 months ago

Updated by Jonathan Bailey 7 months ago · Edited

Updated by Upkeep Bot 7 months ago