os/bluestore: Multiple bdev labels on main block device by aclamk · Pull Request #55374 · ceph/ceph

aclamk · 2024-01-30T13:33:30Z

Corruption of bdev label makes it very hard to recover OSD.
This PR copies bdev label to 4 potential replica places on device:
0(original), 1GB, 10GB, 100GB, 1000GB.

If all replicas do not match system refuses to start.
Fsck (repair mode) is the way to fix it.

This is an alternative version for #53095. Borrows many concepts and some code from it.

Contribution Guidelines

To sign and title your commits, please refer to Submitting Patches to Ceph.
If you are submitting a fix for a stable branch (e.g. "quincy"), please refer to Submitting Patches to Ceph - Backports for the proper workflow.
When filling out the below checklist, you may click boxes directly in the GitHub web UI. When entering or editing the entire PR message in the GitHub web UI editor, you may also select a checklist item by adding an x between the brackets: [x]. Spaces and capitalization matter when checking off items this way.

Checklist

Tracker (select at least one)
- References tracker ticket
- Very recent bug; references commit where it was introduced
- New feature (ticket optional)
- Doc update (no ticket needed)
- Code cleanup (no ticket needed)
Component impact
- Affects Dashboard, opened tracker ticket
- Affects Orchestrator, opened tracker ticket
- No impact that needs to be tracked
Documentation (select at least one)
- Updates relevant documentation
- No doc update is appropriate
Tests (select at least one)
- Includes unit test(s)
- Includes integration test(s)
- Includes bug reproducer
- No tests

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox
jenkins test windows
jenkins test rook e2e

pereman2 · 2024-01-30T16:06:31Z

src/os/bluestore/BlueStore.h

  int _check_or_set_bdev_label(std::string path, uint64_t size, std::string desc,
 			       bool create);
+  int _check_or_set_main_bdev_label(
+    std::string path,


Suggested change

std::string path,

std::string& path,

src/os/bluestore/BlueStore.cc

pereman2 · 2024-01-30T16:26:04Z

src/os/bluestore/BlueStore.cc

+  if (bdev_label_valid_locations.empty()) {
+    _read_main_bdev_label(cct, p, &bdev_label,
+      &bdev_label_valid_locations, &bdev_label_multi, &bdev_label_epoch);
+  }
+  if (!bdev_label_valid_locations.empty()) {
+    bdev_label.meta[key] = value;
+    if (bdev_label_multi) {
+      bdev_label_epoch++;
+      bdev_label.meta["epoch"] = std::to_string(bdev_label_epoch);
+    }
+    int r = _write_bdev_label(cct, p, bdev_label, bdev_label_valid_locations);
+    ceph_assert(r == 0);
  }
-  label.meta[key] = value;
-  r = _write_bdev_label(cct, p, label);
-  ceph_assert(r == 0);
  return ObjectStore::write_meta(key, value);
 }


What if bdev_label_valid_locations.empty() == true?

We need to skip writing to bdev label if no bdev label was available, because we write "type=bluestore" in mkfs before creation of bdev label.
Granted, its weird.

aclamk · 2024-01-31T11:49:46Z

jenkins test make check

github-actions · 2024-02-02T17:33:59Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

src/os/bluestore/BlueStore.h

src/os/bluestore/BlueStore.cc

github-actions · 2024-02-06T14:52:57Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

ifed01 · 2024-02-06T16:31:24Z

src/os/bluestore/BlueStore.cc

+      }
+    }
+    // Mark bits or locations of all bdev labels.
+    for (size_t i = 0; i < bdev_label_positions.size(); i++) {


shouldn't this go over bdev_labels_in_repair instead?

No, we can have bdev labels locations that have proper data, but are in collision with some object.

well, the above comment is a bit confusing then - in fact you mark all the possible bdev locations here. Irrespective of their bluefs usage.

I have completely different comment here. Lets check after push.

src/os/bluestore/BlueStore.cc

aclamk · 2024-02-09T15:22:09Z

jenkins test make check

src/os/bluestore/BlueStore.cc

aclamk · 2024-02-20T14:59:36Z

Early teuthology run
https://pulpito.ceph.com/akupczyk-2024-02-13_18:55:20-rados-wip-aclamk-bs-multi-label-distro-default-smithi/

github-actions · 2024-07-22T12:37:45Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

NitzanMordhai · 2024-08-04T10:03:04Z

Rados approved: https://tracker.ceph.com/projects/rados/wiki/MAIN#httpstrackercephcomissues67266

yuriw · 2024-08-04T15:09:58Z

jenkins test this please

yuriw · 2024-08-04T15:10:56Z

@aclamk this was tested and approved see: https://tracker.ceph.com/issues/67266
pls merge at will

aclamk · 2024-08-06T10:51:11Z

jenkins test api

aclamk · 2024-08-06T12:59:01Z

jenkins test api

aclamk · 2024-08-06T20:10:06Z

jenkins test api

BlueStore now writes its metadata at multiple offset on devices [1]. It means `ceph-volume lvm zap` doesn't remove BlueStore signature altogether. This can confuse ceph-volume when redeploying an OSD on a previously zapped device because there is still old BlueStore metadata on it. ceph-volume should call `ceph-bluestore-tool zap-device` [2] in addition to the existing calls when wiping a device. [1] ceph#55374 [2] ceph#59632 Fixes: https://tracker.ceph.com/issues/68035 Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

BlueStore now writes its metadata at multiple offset on devices [1]. It means `ceph-volume lvm zap` doesn't remove BlueStore signature altogether. This can confuse ceph-volume when redeploying an OSD on a previously zapped device because there is still old BlueStore metadata on it. ceph-volume should call `ceph-bluestore-tool zap-device` [2] in addition to the existing calls when wiping a device. [1] ceph#55374 [2] ceph#59632 Fixes: https://tracker.ceph.com/issues/68035 Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com> (cherry picked from commit dcf7439)

BlueStore now writes its metadata at multiple offset on devices [1]. It means `ceph-volume lvm zap` doesn't remove BlueStore signature altogether. This can confuse ceph-volume when redeploying an OSD on a previously zapped device because there is still old BlueStore metadata on it. ceph-volume should call `ceph-bluestore-tool zap-device` [2] in addition to the existing calls when wiping a device. [1] ceph#55374 [2] ceph#59632 Fixes: https://tracker.ceph.com/issues/68035 Signed-off-by: Guillaume Abrioux <gabrioux@ibm.com>

sergiuiacob1 · 2025-12-19T11:43:48Z

Hello! What is the proper way to remove these labels for ceph v20.2.0 and up? I'm especially targeting a ceph v18 -> v20 migration. I'm currently using this logic:

  # Check if the device's cluster FSID matches the current cluster
  if [ -n "$DEVICE_CLUSTER_FSID" ] && [ "$DEVICE_CLUSTER_FSID" != "$CLUSTER_FSID" ]; then
    echo "WARNING: FSID MISMATCH DETECTED!"
    echo "  Device has ceph_fsid: $DEVICE_CLUSTER_FSID"
    echo "  Current cluster FSID: $CLUSTER_FSID"
    echo "  The device contains an OSD from a DIFFERENT cluster."
    echo "  Wiping the device to prepare for this cluster..."
    
    # Wipe BlueStore labels at all known locations
    # BlueStore stores labels at: beginning, 1GB, 10GB, and near the end
    echo "Wiping BlueStore labels..."
    dd if=/dev/zero of=/dev/ceph-volume bs=1M count=100 conv=fsync 2>/dev/null || true
    dd if=/dev/zero of=/dev/ceph-volume bs=1M count=10 seek=1024 conv=fsync 2>/dev/null || true
    dd if=/dev/zero of=/dev/ceph-volume bs=1M count=10 seek=10240 conv=fsync 2>/dev/null || true
    dd if=/dev/zero of=/dev/ceph-volume bs=1M count=10 seek=102400 conv=fsync 2>/dev/null || true
    # TODO BlueStore also stores a label "near the END" of the device
    
    # Also try wipefs if available
    wipefs -af /dev/ceph-volume 2>/dev/null || true
    
    echo "Device wiped."
    DEVICE_NEEDS_PREPARE=true
  else
    echo "FSID matches current cluster, reusing existing OSD."
  fi

This needs to work on both on VMs (ceph as a docker container) and on K8s clusters (ceph as a pod).

aclamk requested a review from a team as a code owner January 30, 2024 13:33

aclamk requested review from pereman2 and removed request for a team January 30, 2024 13:33

pereman2 reviewed Jan 30, 2024

View reviewed changes

github-actions bot added bluestore common core tests labels Jan 31, 2024

github-actions bot added the needs-rebase label Feb 2, 2024

aclamk requested a review from pereman2 February 4, 2024 10:26

aclamk force-pushed the wip-aclamk-bs-multi-label branch from 5d754bc to a0530b1 Compare February 5, 2024 12:27

github-actions bot removed the needs-rebase label Feb 5, 2024

aclamk force-pushed the wip-aclamk-bs-multi-label branch from a0530b1 to b5356ba Compare February 5, 2024 23:38

ifed01 reviewed Feb 6, 2024

View reviewed changes

github-actions bot added the needs-rebase label Feb 6, 2024

ifed01 reviewed Feb 6, 2024

View reviewed changes

src/os/bluestore/BlueStore.cc Outdated Show resolved Hide resolved

aclamk force-pushed the wip-aclamk-bs-multi-label branch from df3e95d to 06bfcba Compare February 8, 2024 23:55

github-actions bot removed the needs-rebase label Feb 8, 2024

ifed01 reviewed Feb 9, 2024

View reviewed changes

src/os/bluestore/BlueStore.cc Outdated Show resolved Hide resolved

ifed01 reviewed Feb 9, 2024

View reviewed changes

src/os/bluestore/BlueStore.cc Outdated Show resolved Hide resolved

ifed01 reviewed Feb 9, 2024

View reviewed changes

src/os/bluestore/BlueStore.cc Outdated Show resolved Hide resolved

ifed01 reviewed Feb 9, 2024

View reviewed changes

src/os/bluestore/BlueStore.cc Show resolved Hide resolved

src/os/bluestore/BlueStore.cc Show resolved Hide resolved

aclamk force-pushed the wip-aclamk-bs-multi-label branch 2 times, most recently from 603ec0e to aad1fb8 Compare February 13, 2024 14:32

aclamk force-pushed the wip-aclamk-bs-multi-label branch from 4ea3ecb to 811f297 Compare July 22, 2024 12:45

github-actions bot removed the needs-rebase label Jul 22, 2024

aclamk added the aclamk-testing-phoebe bluestore testing label Jul 22, 2024

os/bluestore: Fix unbalanced new[]/delete[] in read_bdev_label functions

64f6b5d

Signed-off-by: Adam Kupczyk <akupczyk@ibm.com>

ljflores added the wip-yuri6-testing label Jul 30, 2024

yuriw added TESTED ready-to-merge and removed wip-yuri6-testing labels Aug 4, 2024

aclamk merged commit 7bcb68e into ceph:main Aug 7, 2024

aclamk mentioned this pull request Aug 9, 2024

squid: os/bluestore: Multiple bdev labels on main block device #59106

Merged

14 tasks

guits mentioned this pull request Sep 11, 2024

ceph-volume: add call to ceph-bluestore-tool zap-device #59742

Merged

guits mentioned this pull request Sep 25, 2024

BlueStore now replicates its signatures util-linux/util-linux#3219

Closed

sp98 mentioned this pull request Mar 19, 2025

Update Cluster clean up doc for zapping v19.2.1 disks correctly rook/rook#15545

Closed

Conversation

aclamk commented Jan 30, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Contribution Guidelines

Checklist

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

aclamk commented Jan 31, 2024

Uh oh!

github-actions bot commented Feb 2, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions bot commented Feb 6, 2024

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

aclamk commented Feb 9, 2024

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

aclamk commented Feb 20, 2024

Uh oh!

github-actions bot commented Jul 22, 2024

Uh oh!

NitzanMordhai commented Aug 4, 2024

Uh oh!

yuriw commented Aug 4, 2024

Uh oh!

yuriw commented Aug 4, 2024

Uh oh!

aclamk commented Aug 6, 2024

Uh oh!

aclamk commented Aug 6, 2024

Uh oh!

aclamk commented Aug 6, 2024

Uh oh!

sergiuiacob1 commented Dec 19, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

aclamk commented Jan 30, 2024 •

edited

Loading