Project

General

Profile

Actions
Copy link

Bug #69691

closed

bluestore: __ceph_assert_fail for BlueStore::expand_devices

Added by Shraddha Agrawal about 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
-
Target version:
-
% Done:

0%

Source:
Backport:
squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
61470
Tags (freeform):
backport_processed
Merge Commit:
7fcd9cbac04f022c8a95dfed302e308881bc1da4
Fixed In:
v19.3.0-7350-g7fcd9cbac0
Released In:
v20.2.0~1199
Upkeep Timestamp:
2025-11-01T01:29:33+00:00
Tags:
main-failures

Description

In the job /a/skanta-2024-10-24_23:59:35-rados-wip-bharath3-testing-2024-10-23-1509-distro-default-smithi/7965760, we see the following error:

2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr:
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: ceph version 19.3.0-5721-g0cc901e6 (0cc901e6c3bf85cea9dcd864be72472a5942b7dd) squid (dev)
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x121) [0x7fd064b8d73f]
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18d8fe) [0x7fd064b8d8fe]
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 3: (BlueStore::expand_devices(std::ostream&)+0x9d3) [0x55899c337923]
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 4: main()
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 5: /lib64/libc.so.6(+0x29590) [0x7fd064229590]
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 6: __libc_start_main()
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: 7: _start()
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr:
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr:  -496> 2024-10-25T00:46:57.850+0000 7fd06440c9c0 -1 *** Caught signal (Aborted) **
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: in thread 7fd06440c9c0 thread_name:ceph-bluestore-
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr:
2024-10-25T00:46:57.859 INFO:tasks.workunit.client.0.smithi137.stderr: ceph version 19.3.0-5721-g0cc901e6 (0cc901e6c3bf85cea9dcd864be72472a5942b7dd) squid (dev)
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 1: /lib64/libc.so.6(+0x3e6f0) [0x7fd06423e6f0]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 2: /lib64/libc.so.6(+0x8b99c) [0x7fd06428b99c]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 3: raise()
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 4: abort()
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 5: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x17b) [0x7fd064b8d799]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 6: /usr/lib64/ceph/libceph-common.so.2(+0x18d8fe) [0x7fd064b8d8fe]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 7: (BlueStore::expand_devices(std::ostream&)+0x9d3) [0x55899c337923]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 8: main()
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 9: /lib64/libc.so.6(+0x29590) [0x7fd064229590]
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 10: __libc_start_main()
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: 11: _start()
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr: NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this.
2024-10-25T00:46:57.860 INFO:tasks.workunit.client.0.smithi137.stderr:
2024-10-25T00:46:57.988 INFO:tasks.workunit.client.0.smithi137.stderr:/home/ubuntu/cephtest/clone.client.0/qa/standalone/osd/osd-bluefs-volume-ops.sh: line 19: 116520 Aborted                 (core dumped) ceph-bluestore-tool --path $dir/1 bluefs-bdev-expand

We see a __ceph_assert_fail for BlueStore::expand_devices,

This was first seen as part of https://tracker.ceph.com/issues/68711. This was likely introduced in the PR: https://github.com/ceph/ceph/pull/60363 which made changes to the BlueStore::expand_devices(ostream& out) function.

Related issues 1 (0 open — 1 closed)

Copied to bluestore - Backport #69829: squid: bluestore: __ceph_assert_fail for BlueStore::expand_devicesResolvedActions
Actions
Copy link
 #1

Updated by Aishwarya Mathuria about 1 year ago

  • Tags set to main-failures
Actions
Copy link
 #2

Updated by Aishwarya Mathuria about 1 year ago

/a/skanta-2025-01-26_07:44:24-rados-wip-bharath9-testing-2025-01-25-0527-distro-default-smithi/8094187

25-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: in thread 7ff3bcaf1c00 thread_name:ceph-bluestore-
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr:2025-01-26T08:37:06.776+0000 7ff3bcaf1c00 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7084-gf511daef/rpm/el9/BUILD/ceph-19.3.0-7084-gf511daef/src/os/bluestore/BlueStore.cc: In function 'int BlueStore::expand_devices(std::ostream&)' thread 7ff3bcaf1c00 time 2025-01-26T08:37:06.777092+0000
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr:/home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos9/DIST/centos9/MACHINE_SIZE/gigantic/release/19.3.0-7084-gf511daef/rpm/el9/BUILD/ceph-19.3.0-7084-gf511daef/src/os/bluestore/BlueStore.cc: 8987: FAILED ceph_assert(r == 0)
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr:
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: ceph version 19.3.0-7084-gf511daef (f511daef9e5fd5e5dec6dd2bb9bf3b44b4c7c4e2) squid (dev)
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x11e) [0x7ff3bdf8c63c]
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 2: /usr/lib64/ceph/libceph-common.so.2(+0x18c7fb) [0x7ff3bdf8c7fb]
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 3: (BlueStore::expand_devices(std::ostream&)+0x911) [0x5623a975fdd1]
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 4: main()
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 5: /lib64/libc.so.6(+0x295d0) [0x7ff3bd6295d0]
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 6: __libc_start_main()
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr: 7: _start()
2025-01-26T08:37:06.781 INFO:tasks.workunit.client.0.smithi088.stderr:
Actions
Copy link
 #3

Updated by Laura Flores about 1 year ago

/a/yuriw-2025-01-31_15:46:33-rados-wip-yuri5-testing-2025-01-30-1311-distro-default-smithi/8106708

Actions
Copy link
 #5

Updated by Adam Kupczyk about 1 year ago

  • Status changed from New to Resolved
  • Backport set to squid
  • Pull request ID set to 61470
Actions
Copy link
 #6

Updated by Adam Kupczyk about 1 year ago

  • Status changed from Resolved to Pending Backport
Actions
Copy link
 #7

Updated by Upkeep Bot about 1 year ago

  • Copied to Backport #69829: squid: bluestore: __ceph_assert_fail for BlueStore::expand_devices added
Actions
Copy link
 #8

Updated by Upkeep Bot about 1 year ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #9

Updated by Shraddha Agrawal about 1 year ago

/a/skanta-2025-01-28_15:23:42-rados-wip-bharath11-testing-2025-01-27-1602-distro-default-smithi/8098964

Actions
Copy link
 #10

Updated by Shraddha Agrawal about 1 year ago

/a/skanta-2025-02-05_10:08:28-rados-wip-bharath3-testing-2025-02-03-2127-distro-default-smithi/8115708

Actions
Copy link
 #11

Updated by Aishwarya Mathuria about 1 year ago

/a/lflores-2025-02-07_20:42:43-rados-wip-yuri2-testing-2025-01-31-2325-distro-default-smithi/8120802

Actions
Copy link
 #12

Updated by Igor Fedotov about 1 year ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #13

Updated by Kamoltat (Junior) Sirivadhna about 1 year ago

This run was created before the fix, so just pasting the failure here:
yuriw-2025-02-07_15:56:23-rados-wip-yuri6-testing-2025-02-04-1046-distro-default-smithi/8120707

Actions
Copy link
 #14

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to 7fcd9cbac04f022c8a95dfed302e308881bc1da4
  • Fixed In set to v19.3.0-7350-g7fcd9cbac04
  • Upkeep Timestamp set to 2025-07-10T00:35:33+00:00
Actions
Copy link
 #15

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-7350-g7fcd9cbac04 to v19.3.0-7350-g7fcd9cbac0
  • Upkeep Timestamp changed from 2025-07-10T00:35:33+00:00 to 2025-07-14T19:37:30+00:00
Actions
Copy link
 #16

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~1199
  • Upkeep Timestamp changed from 2025-07-14T19:37:30+00:00 to 2025-11-01T01:29:33+00:00
Actions
Copy link

Also available in: Atom PDF