Project

General

Profile

Actions
Copy link

Bug #67911

closed

ceph-objectstore-tool fails on export from bluefs label corruption

Added by Laura Flores over 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Adam Kupczyk
Target version:
-
% Done:

0%

Source:
Backport:
squid
Regression:
No
Severity:
2 - major
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
59850
Tags (freeform):
backport_processed
Merge Commit:
5934c74b733ada90735d64601e9414b4fc4910ce
Fixed In:
v19.3.0-5164-g5934c74b73
Released In:
v20.2.0~1940
Upkeep Timestamp:
2025-11-01T01:30:50+00:00

Description

/a/yuriw-2024-08-29_14:15:40-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7880224

2024-08-29T19:00:33.993 DEBUG:teuthology.orchestra.run.smithi097:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell 'osd.*' injectargs --osd_enable_op_tracker=false
2024-08-29T19:00:33.994 INFO:tasks.ceph.osd.1.smithi097.stderr:2024-08-29T19:00:33.992+0000 7f520749b640 -1 received  signal: Hangup from /usr/bin/python3 /bin/daemon-helper kill ceph-osd -f --cluster ceph -i 1  (PID: 22520) UID: 0
2024-08-29T19:00:33.995 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 30 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.5.asok dump_ops_in_flight
2024-08-29T19:00:34.012 INFO:teuthology.orchestra.run.smithi097.stderr:Mount failed with '2024-08-29T19:00:34.011+0000 7fa84f617580 -1 bluestore(/var/lib/ceph/osd/ceph-0) _mount fsck found 1 errors
2024-08-29T19:00:34.012 INFO:teuthology.orchestra.run.smithi097.stderr:(5) Input/output error'
2024-08-29T19:00:34.023 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-08-29T19:00:34.024 INFO:tasks.thrashosds.thrasher:Traceback (most recent call last):
  File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/ceph_manager.py", line 192, in wrapper
    return func(self)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/ceph_manager.py", line 1435, in _do_thrash
    self.choose_action()()
  File "/home/teuthworker/src/github.com_ceph_ceph-c_1ef864504b8875c83ee6c2c5fedc13315bebf7f5/qa/tasks/ceph_manager.py", line 445, in kill_osd
    raise Exception("ceph-objectstore-tool: " 
Exception: ceph-objectstore-tool: export failure with status 1

/a/yuriw-2024-08-29_14:15:40-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7880224/remote/smithi097/log/objectstore_tool.56823.log.gz

2024-08-29T19:00:33.513+0000 7fa844f3a640 20 bluestore(/var/lib/ceph/osd/ceph-0) deferred_try_submit 0 osrs, 0 txcs
2024-08-29T19:00:33.513+0000 7fa844f3a640  5 bluestore.MempoolThread(0x55e9f6114bf0) _resize_shards cache_size: 2844627089 kv_alloc: 1191182336 kv_used: 46208 kv_onode_alloc: 234881024 kv_onode_used: 26448 meta_alloc: 1124073472 meta_used: 680 data_alloc: 218103808 data_used: 0
2024-08-29T19:00:33.563+0000 7fa844f3a640 20 bluestore.MempoolThread(0x55e9f6114bf0) _resize_shards cache_size: 2844627089 kv_alloc: 1191182336 kv_used: 46208 kv_onode_alloc: 234881024 kv_onode_used: 26448 meta_alloc: 1124073472 meta_used: 680 data_alloc: 218103808 data_used: 0
2024-08-29T19:00:33.576+0000 7fa84f617580 -1 bluestore(/var/lib/ceph/osd/ceph-0) fsck error: bdev label at 0x40000000 corrupted
2024-08-29T19:00:33.577+0000 7fa84f617580 10 bluefs foreach_block_extents bdev 1
2024-08-29T19:00:33.578+0000 7fa84f617580  1 bluestore(/var/lib/ceph/osd/ceph-0) fsck bdev label at 0x40000000taken by bluefs, cannot be fixed

Related issues 1 (0 open — 1 closed)

Copied to bluestore - Backport #68222: squid: ceph-objectstore-tool fails on export from bluefs label corruptionResolvedAdam KupczykActions
Actions
Copy link
 #1

Updated by Laura Flores over 1 year ago

  • Project changed from RADOS to bluestore
  • Subject changed from ceph-objectstoretool fails on export from bluefs label corruption to ceph-objectstore-tool fails on export from bluefs label corruption
Actions
Copy link
 #2

Updated by Laura Flores over 1 year ago

  • Backport set to squid
Actions
Copy link
 #3

Updated by Laura Flores over 1 year ago

/a/yuriw-2024-08-29_14:15:40-rados-wip-yuri8-testing-2024-08-28-1632-squid-distro-default-smithi/7880330

Actions
Copy link
 #4

Updated by Igor Fedotov over 1 year ago

  • Assignee set to Adam Kupczyk

@Adam Kupczyk - mind taking a look?

Actions
Copy link
 #5

Updated by Adam Kupczyk over 1 year ago

  • Backport changed from squid to reef
  • Severity changed from 3 - minor to 2 - major
  • Pull request ID set to 59850
Actions
Copy link
 #6

Updated by Adam Kupczyk over 1 year ago

  • Backport changed from reef to squid
Actions
Copy link
 #7

Updated by Igor Fedotov over 1 year ago

  • Status changed from New to Fix Under Review
Actions
Copy link
 #8

Updated by Adam Kupczyk over 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions
Copy link
 #9

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #68222: squid: ceph-objectstore-tool fails on export from bluefs label corruption added
Actions
Copy link
 #10

Updated by Upkeep Bot over 1 year ago

  • Tags (freeform) set to backport_processed
Actions
Copy link
 #11

Updated by Kamoltat (Junior) Sirivadhna over 1 year ago

yuriw-2024-10-16_22:34:49-rados-wip-yuri10-testing-2024-10-15-1007-distro-default-smithi/7952541

Actions
Copy link
 #12

Updated by Igor Fedotov about 1 year ago

  • Status changed from Pending Backport to Resolved
Actions
Copy link
 #13

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to 5934c74b733ada90735d64601e9414b4fc4910ce
  • Fixed In set to v19.3.0-5164-g5934c74b733
  • Upkeep Timestamp set to 2025-07-10T13:46:05+00:00
Actions
Copy link
 #14

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-5164-g5934c74b733 to v19.3.0-5164-g5934c74b73
  • Upkeep Timestamp changed from 2025-07-10T13:46:05+00:00 to 2025-07-14T20:10:59+00:00
Actions
Copy link
 #15

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~1940
  • Upkeep Timestamp changed from 2025-07-14T20:10:59+00:00 to 2025-11-01T01:30:50+00:00
Actions
Copy link

Also available in: Atom PDF