mgr/rook: implement `orch device zap` in rook orchestrator by josephsawaya · Pull Request #43138 · ceph/ceph

josephsawaya · 2021-09-10T18:43:32Z

This PR implements orch device zap by creating a Batch job that
creates a pod on the target host that mounts the /dev directory and
runs ceph-volume lvm zap --destroy <path>

Signed-off-by: Joseph Sawaya jsawaya@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard cephadm
jenkins test api
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

leseb

You also need to address raw OSDs.

github-actions · 2021-09-13T14:51:11Z

This pull request can no longer be automatically merged: a rebase is needed and changes have to be manually resolved

jeffvance · 2021-09-13T18:44:05Z

src/pybind/mgr/rook/rook_cluster.py

+                )
+            )
+        )
+        self.batchV1_api.create_namespaced_job('rook-ceph', body)


no err/status returned?

josephsawaya · 2021-09-13T20:29:48Z

You also need to address raw OSDs.

From what I can tell ceph-volume lvm zap is the best option right now, there is no ceph-volume raw zap and it doesn't seem like there's any specific command for zapping raw devices specifically in ceph-volume

liewegas · 2021-09-13T20:32:31Z

You also need to address raw OSDs.

From what I can tell ceph-volume lvm zap is the best option right now, there is no ceph-volume raw zap and it doesn't seem like there's any specific command for zapping raw devices specifically in ceph-volume

sort out zapping raw devices from ceph-volume is on the todo list!

leseb · 2021-09-14T07:01:10Z

You also need to address raw OSDs.

From what I can tell ceph-volume lvm zap is the best option right now, there is no ceph-volume raw zap and it doesn't seem like there's any specific command for zapping raw devices specifically in ceph-volume

sort out zapping raw devices from ceph-volume is on the todo list!

Yeah but for now we could just dd the first few MiB of the disk if it's a raw OSD so that the PR is complete. Up to you.

jmolmo · 2021-09-14T09:38:30Z

Several doubts here, probably is just that for me is not clear the roadmap ahead...... for the moment , we only can create OSDs using PVs, and we do not have information about physical details of the devices, and we do not now if an specific device is used in a PV and for several PVCs.
It is ok to manage PVs (and do not show any device information) in commands like "ceph orch device ls" and at the same time provide a command to destroy a physical device? It is ok to launch the job without any verification about the use of the device?

jmolmo · 2021-09-14T09:43:58Z

I think now i understand ... this is just to make possible the theutology test:
#43139
... but probably highly dangerous outside theutology

josephsawaya · 2021-09-14T13:51:14Z

I think now i understand ... this is just to make possible the theutology test:
#43139
... but probably highly dangerous outside theutology

Yeah for now it looks like this should just be used in teuthology.

You also need to address raw OSDs.

From what I can tell ceph-volume lvm zap is the best option right now, there is no ceph-volume raw zap and it doesn't seem like there's any specific command for zapping raw devices specifically in ceph-volume

sort out zapping raw devices from ceph-volume is on the todo list!

Yeah but for now we could just dd the first few MiB of the disk if it's a raw OSD so that the PR is complete. Up to you.

I just went ahead and changed it to use dd in the case of a raw device

liewegas · 2021-10-13T22:43:52Z

jenkins test make check

leseb · 2021-10-13T22:52:12Z

src/pybind/mgr/rook/module.py

+            self.rook_cluster.create_zap_job(host, path)
+        except Exception as e:
+            logging.error(e)
+            return OrchResult(None, "Unable to zap device" + str(e.with_traceback(None)))


Suggested change

return OrchResult(None, "Unable to zap device" + str(e.with_traceback(None)))

return OrchResult(None, "Unable to zap device " + str(e.with_traceback(None)))

Or is OrchResult printing nicely already?

leseb · 2021-10-13T22:53:35Z

src/pybind/mgr/rook/rook_cluster.py

+                                name="device-zap",
+                                image="rook/ceph:master",
+                                command=["bash"],
+                                args=["-c", f"ceph-volume raw list | grep {path} && dd if=/dev/zero of=\"{path}\" bs=1M count=100 oflag=direct,dsync || ceph-volume lvm zap --destroy {path}"],


Suggested change

args=["-c", f"ceph-volume raw list | grep {path} && dd if=/dev/zero of=\"{path}\" bs=1M count=100 oflag=direct,dsync || ceph-volume lvm zap --destroy {path}"],

args=["-c", f"ceph-volume raw list {path} && dd if=/dev/zero of=\"{path}\" bs=1M count=100 oflag=direct,dsync || ceph-volume lvm zap --destroy {path}"],

This should be the same. IIRC the list supports arguments.

Also, I think count=1 is probably more than enough since the bluestore signature is 60 bytes only.

One more thing, lvm zap supports OSD FSID and/or OSD ID as a parameter, I wonder if it's safer to use instead of passing the disk, just a thought.

liewegas · 2021-10-14T14:21:18Z

rook/__init__.py:5: note: In module imported here:
rook/module.py: note: In member "zap_device" of class "RookOrchestrator":
rook/module.py:452: error: Argument 2 to "OrchResult" has incompatible type "str"; expected "Optional[Exception]"

liewegas · 2021-10-18T17:58:48Z

jenkins test make check

liewegas · 2021-10-18T19:02:25Z

src/pybind/mgr/rook/rook_cluster.py

            )
            ret.append(spec)
        return ret
+    def create_zap_job(self, host: str, path: str) -> None:


missing newline above

This commit implements orch device zap by creating a pod on the target host that mounts the /dev directory and runs either overwrites the first few blocks of the device with zeros if it's a raw device or if it's not a raw device it will use `ceph-volume lvm zap`. Signed-off-by: Joseph Sawaya <jsawaya@redhat.com>

liewegas · 2021-10-18T21:45:42Z

jenkins test api

sebastian-philipp · 2021-10-20T16:07:05Z

jenkins test api

josephsawaya requested a review from a team as a code owner September 10, 2021 18:43

github-actions bot added orchestrator pybind rook labels Sep 10, 2021

josephsawaya mentioned this pull request Sep 10, 2021

qa/tasks/rook: test reapplication of drive groups stored in mgr #43139

Merged

3 tasks

leseb reviewed Sep 13, 2021

View reviewed changes

github-actions bot added the needs-rebase label Sep 13, 2021

josephsawaya force-pushed the wip-mgr-rook-device-zap branch from 595d1cd to 4dda0b1 Compare September 13, 2021 15:14

github-actions bot removed the needs-rebase label Sep 13, 2021

jeffvance reviewed Sep 13, 2021

View reviewed changes

src/pybind/mgr/rook/rook_cluster.py

)

)

)

self.batchV1_api.create_namespaced_job('rook-ceph', body)

Copy link

jeffvance Sep 13, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no err/status returned?

josephsawaya force-pushed the wip-mgr-rook-device-zap branch from 4dda0b1 to 30342b3 Compare September 14, 2021 13:47

josephsawaya force-pushed the wip-mgr-rook-device-zap branch from 30342b3 to 2840b01 Compare September 17, 2021 13:28

liewegas requested a review from leseb October 13, 2021 21:10

liewegas added the wip-sage-testing label Oct 13, 2021

leseb reviewed Oct 13, 2021

View reviewed changes

josephsawaya force-pushed the wip-mgr-rook-device-zap branch from 2840b01 to d4107b6 Compare October 14, 2021 21:35

liewegas reviewed Oct 18, 2021

View reviewed changes

josephsawaya force-pushed the wip-mgr-rook-device-zap branch from d4107b6 to 22631ec Compare October 18, 2021 19:06

liewegas approved these changes Oct 18, 2021

View reviewed changes

leseb approved these changes Oct 19, 2021

View reviewed changes

liewegas merged commit 78f8f78 into ceph:master Oct 20, 2021

	return OrchResult(None, "Unable to zap device" + str(e.with_traceback(None)))
	return OrchResult(None, "Unable to zap device " + str(e.with_traceback(None)))

	args=["-c", f"ceph-volume raw list \| grep {path} && dd if=/dev/zero of=\"{path}\" bs=1M count=100 oflag=direct,dsync \|\| ceph-volume lvm zap --destroy {path}"],
	args=["-c", f"ceph-volume raw list {path} && dd if=/dev/zero of=\"{path}\" bs=1M count=100 oflag=direct,dsync \|\| ceph-volume lvm zap --destroy {path}"],

Conversation

josephsawaya commented Sep 10, 2021

Checklist

Uh oh!

leseb left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions bot commented Sep 13, 2021

Uh oh!

jeffvance Sep 13, 2021

Choose a reason for hiding this comment

Uh oh!

josephsawaya commented Sep 13, 2021

Uh oh!

liewegas commented Sep 13, 2021

Uh oh!

leseb commented Sep 14, 2021

Uh oh!

jmolmo commented Sep 14, 2021

Uh oh!

jmolmo commented Sep 14, 2021

Uh oh!

josephsawaya commented Sep 14, 2021

Uh oh!

liewegas commented Oct 13, 2021

Uh oh!

leseb Oct 13, 2021

Choose a reason for hiding this comment

Uh oh!

leseb Oct 13, 2021

Choose a reason for hiding this comment

Uh oh!

leseb Oct 13, 2021

Choose a reason for hiding this comment

Uh oh!

leseb Oct 13, 2021

Choose a reason for hiding this comment

Uh oh!

liewegas commented Oct 14, 2021

Uh oh!

liewegas commented Oct 18, 2021

Uh oh!

liewegas Oct 18, 2021

Choose a reason for hiding this comment

Uh oh!

liewegas commented Oct 18, 2021

Uh oh!

sebastian-philipp commented Oct 20, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants