nvmeof: add examples and deployment files by nixpanic · Pull Request #5641 · ceph/ceph-csi

nixpanic · 2025-10-09T09:32:05Z

Examples and deployment files for NVMe-oF. These are used with the e2e testing
as well.

See-also: #5672

nixpanic · 2025-10-09T09:32:44Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-10-09T10:38:49Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-10-09T15:41:54Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-10-10T08:05:17Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-10T08:06:25Z

/test ci/centos/mini-e2e/k8s-1.33

nixpanic · 2025-10-10T08:49:46Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-10-10T09:43:47Z

/test ci/centos/mini-e2e/k8s-1.33/nvmeof

nixpanic · 2025-10-10T09:43:59Z

/test ci/centos/mini-e2e/k8s-1.32

nixpanic · 2025-10-10T11:32:12Z

Deploying the NVMe-oF gateway fails because the ceph nvme-gw create ... command was not found. Maybe this has been added to newer Ceph version, and isn't available in v19.2.2.

Log from the init-container that constructs the configuration and announces the gateway in the Ceph environment:

  + ceph nvme-gw create ceph-nvmeof-gateway-59f798f746-rjv9h nvmeofpool rook-ceph
  no valid command found; 10 closest matches:
  pg stat
  pg getmap
  pg dump [<dumpcontents:all|summary|sum|delta|pools|osds|pgs|pgs_brief>...]
  pg dump_json [<dumpcontents:all|summary|sum|pools|osds|pgs>...]
  pg dump_pools_json
  pg ls-by-pool <poolstr> [<states>...]
  pg ls-by-primary <id|osd.id> [<pool:int>] [<states>...]
  pg ls-by-osd <id|osd.id> [<pool:int>] [<states>...]
  pg ls [<pool:int>] [<states>...]
  pg dump_stuck [<stuckops:inactive|unclean|stale|undersized|degraded>...] [<threshold:int>]
  Error EINVAL: invalid command

nixpanic · 2025-10-10T11:36:43Z

/test ci/centos/mini-e2e/k8s-1.34

nixpanic · 2025-10-10T11:50:43Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-10T12:18:12Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-10T13:40:33Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-10T15:23:35Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-10T15:59:54Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-12T12:27:27Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-13T06:57:49Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-13T08:39:43Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

nixpanic · 2025-10-13T10:56:52Z

/test ci/centos/mini-e2e/k8s-1.34/nvmeof

gadididi

LGTM!
small nits\questions

gadididi · 2026-02-08T08:46:34Z

+  listeners: |
+    [{
+      "address": "10.242.64.32",
+      "port": "4420",
+      "hostname": "cephnvme-vm9"
+    }]


maybe we should write under comment the alternative way (AutoListener. and write it will be supported from ceph tentacle?

UPDATE:
Aviv does not want to release the manually option at all, because he does not want users will start use this (uneasy) approach..

Autolisteners are not available yet, right? Needs a newer Ceph version, new gateway and updated Ceph-CSI.

This is what we have now, and it works for Ceph clusters that run their gateways outside of the Kubernetes cluster.

We can update the examples later on, if we know when or what version combination works better.

But hopefully we get a way that deploys/configures the gateway in a stable fashion with Rook. Services and Cluster-IP addresses without specifying IP-addresses in the listerners section would be a clean approach too.

ok sounds good, thanks

gadididi · 2026-02-08T08:49:57Z

+metadata:
+  name: "ceph-csi-config"


We can add here in metadata entry- the kms type (under comment)
//"encryptionKMSType": "metadata"
(If someone works with dh-chap he must add it)

ideally the PR that adds dh-chap would add the option here. If that feature gets merged before this, we can include it here.

gadididi · 2026-02-08T08:56:56Z

+          # FIXME: delete imagePullPolicy, or set to "IfNotPresent"
+          imagePullPolicy: "Always"


gadididi · 2026-02-08T09:00:30Z

@@ -0,0 +1,12 @@
+---


maybe we can add another PVC example with
volumeAttributesClassName property, and its value will be default

Not all kubernetes versions we are testing with, support that yet. Creating the PVC with an old Kubernetes version would fail.

gadididi · 2026-02-08T09:01:10Z

+
+  # subsystemNQN is the NQN of the subsystem where all volumes belong to
+  subsystemNQN: "nqn.2025-08.io.ceph:k8s-ceph-csi"
+


maybe add dh-chap option under comment?

see #5641 (comment)

gadididi · 2026-02-08T11:48:53Z


 	e2eGatewaySecurityContext = e2eGatewayPath + "scc.yaml"
 	e2eGatewayServiceAccount  = e2eGatewayPath + "serviceaccount.yaml"
 	e2eGatewayConfig          = e2eGatewayPath + "config.yaml"


inside the e2e/nvmeof/config.yaml file I think we should add this line:
enable_monitor_client = True in the GW configuration. without monitor client the GW cannot be deployed

Well, it works now, I'd rather change as little as possible at the moment. Hopefully we can use Rook to deploy the gateway soon.

nixpanic · 2026-02-10T09:32:02Z

+		// After a failure, the deployment is automatically removed. The gateway Pod
+		// is not available anymore when the e2e job gathers all the logs. Record
+		// the logs in the job for now (the way of deploying will move to Rook in
+		// the future anyway).
+		for _, container := range []string{"generate-minimal-ceph-conf", "nvmeof-gateway"} {
+			logs, _ := frameworkPod.GetPodLogs(context.TODO(), f.ClientSet, rookNamespace, pod, container)
+			framework.Logf("Logs from the %q container of the NVMe-oF gateway:\n%s", container, logs)
+		}


I hope we can drop all the manual deploying of the gateway and rely on Rook soon.

gadididi · 2026-02-08T13:00:57Z

+	Expect(err).ShouldNot(HaveOccurred())
+	Expect(pods.Items).Should(HaveLen(1))
+
+	return pods.Items[0].Name, pods.Items[0].Status.PodIP


maybe return list of ip port , depending on how many GW you deployed in e2e/nvmeof/deployment.yaml

gadididi · 2026-02-09T08:03:46Z

+		Port     int    `json:"port"`
+	}{
+		{
+			Hostname: gwHost, // FIXME: is this required?


remove the comment

gadididi · 2026-02-09T08:14:21Z

+			validateRBDImageCount(f, 1, nvmeofPool)
+			validateOmapCount(f, 1, rbdType, nvmeofPool, volumesType)
+


maybe in the future we can add another pod nvmeog-gw-toolbox and then validate the listener\host\subsystem\namespace were created, what do you think?

Sure, when we add tests that validate the state of the gateway, we should deploy the additional toolbox too.

Madhu-1

@nixpanic is the plan to add all the yamls here and later migrate to the csi-operator once its ready?

Madhu-1 · 2026-02-09T09:38:48Z

+metadata:
+  name: ceph-nvmeof-nodeplugin
+rules:
+  - apiGroups: [""]


Are these RBAC's copy from the RBD driver or nvme needs these permissions as well?

the NVMe implementation calls some RBD functionality. It may not need all permissions at the moment, but new features get added, so the permissions need to be updated at that time too. Instead of trying to remove permissions that get added later, I just kept them.

Madhu-1 · 2026-02-09T09:40:20Z

+            - mountPath: /csi-logs
+              name: logs-dir
+        - args:
+            - --v=5


log level 1 like in deployment?

Madhu-1 · 2026-02-09T09:41:13Z

+	return s
+}
+
+// parse splilts the version string into sepate pieces.


splilts to splits sepate to separate

Madhu-1 · 2026-02-09T09:43:49Z

 	flag.BoolVar(&testNBD, "test-nbd", false, "test rbd csi driver with rbd-nbd mounter")
 	flag.BoolVar(&testNFS, "test-nfs", false, "test nfs csi driver")
-	flag.BoolVar(&testNVMeoF, "test-nvmeof", false, "test nvmeof csi driver")
+	flag.BoolVar(&testNVMeoF, "test-nvmeof", true, "test nvmeof csi driver")


this should be set to false as we dont deploy-nvme by default?

testing should be done by default. When no support for the nvme-gateway is detected, the tests are skipped. This option makes it possible to explicitly disable the testing, if needed for some reason.

Madhu-1 · 2026-02-09T09:43:55Z

@@ -0,0 +1,249 @@
+/*
+Copyright 2025 The Ceph-CSI Authors.


Madhu-1 · 2026-02-09T09:45:23Z

+	}
+	rawListeners, err := json.Marshal(listeners)
+	Expect(err).ShouldNot(HaveOccurred())
+	sc.Parameters["listeners"] = string(rawListeners)


i thought we were going to use auto listeners as the listeners will have problem if the gateway pod restarts, this will leave inconcistent test results

Autolisteners are not available yet. Ideally we do not need to provide IP-addresses when the gateway gets deployed by Rook. But how that will be done, is something we need to figure out.

@Madhu-1
the AutoListener will not resolve the nvme connection issue (from the nodesever side) . it just will make easier life for the administrator . No need to re-edit the SC every time the GW pod restarts with the IPs..

the connection issue is still open without solution. hopefully soon will be resolved

Madhu-1 · 2026-02-09T09:48:28Z

+  csi.storage.k8s.io/controller-expand-secret-name: csi-nvmeof-secret
+  csi.storage.k8s.io/controller-expand-secret-namespace: default
+  csi.storage.k8s.io/node-stage-secret-name: csi-nvmeof-secret
+  csi.storage.k8s.io/node-stage-secret-namespace: default


publish secrets are not required?

No, maybe later when dh-chap is supported.

Madhu-1 · 2026-02-09T09:49:46Z

@@ -0,0 +1,13 @@
+---
+# apiVersion: storage.k8s.io/v1beta1 for k8s version 1.33


IMO one class example is enough with commented parameters, we dont need to add multiple files to show different parameters

I think practical examples are valuable for users. It makes it easier to understand that they should create multiple VolumeAttributesClasses.

Madhu-1 · 2026-02-09T09:50:50Z

+	sc.Parameters["csi.storage.k8s.io/provisioner-secret-name"] = nvmeofProvisionerSecretName
+
+	sc.Parameters["csi.storage.k8s.io/controller-publish-secret-namespace"] = cephCSINamespace
+	sc.Parameters["csi.storage.k8s.io/controller-publish-secret-name"] = nvmeofProvisionerSecretName


expand secrets are missing here?

Not really missing, there is no expand volume test yet. I'll add them to keep everything a little more complete.

nixpanic · 2026-02-10T09:32:02Z

+		// After a failure, the deployment is automatically removed. The gateway Pod
+		// is not available anymore when the e2e job gathers all the logs. Record
+		// the logs in the job for now (the way of deploying will move to Rook in
+		// the future anyway).
+		for _, container := range []string{"generate-minimal-ceph-conf", "nvmeof-gateway"} {
+			logs, _ := frameworkPod.GetPodLogs(context.TODO(), f.ClientSet, rookNamespace, pod, container)
+			framework.Logf("Logs from the %q container of the NVMe-oF gateway:\n%s", container, logs)
+		}


I hope we can drop all the manual deploying of the gateway and rely on Rook soon.

nixpanic · 2026-02-10T09:33:43Z

@nixpanic is the plan to add all the yamls here and later migrate to the csi-operator once its ready?

Yes, ceph-csi-operator isn't ready yet, so we have to do this for the time being.

Add example of storage class and pvc, need to modify storage class before use! Signed-off-by: Niels de Vos <ndevos@ibm.com> Signed-off-by: gadi-didi <gadi.didi@ibm.com>

The deployment files make it possible to start the NVMe-oF controller and nodeplugin by the e2e suite. Co-authored-by: gadi-didi <gadi.didi@ibm.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

Create a PVC and wait for it to get Bound. This does not exersice the node-plugin yet, only the controller/provisioner. Signed-off-by: Niels de Vos <ndevos@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

Recent versions of Rook use Ceph-CSI Operator by default. There is no need to have Ceph-CSI deployed by Rook, as our CI deploys it from the PR that is being tested. Signed-off-by: Niels de Vos <ndevos@ibm.com>

ceph-volume in Tentacle (v20) uses udev data to detect some properties of the potential OSDs. Inside the minikube cluster (running with Podman), there is no udev. Mounting the host udev runtime data into the containers makes ceph-volume happy. Signed-off-by: Niels de Vos <ndevos@ibm.com>

Signed-off-by: Niels de Vos <ndevos@ibm.com>

Add example of volumeAttributesClass yaml files. it is possible to use in with PVC, by using the field volumeAttributesClassName. Co-authored-by: Niels de Vos <ndevos@ibm.com> Signed-off-by: gadi-didi <gadi.didi@ibm.com>

Ceph Tentacle is required for deploying the NVMe-oF gateway. Earlier versions do not support the gateway. Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic · 2026-02-10T10:22:57Z

@gadididi and @Madhu-1 , thanks for the review and comments. Please have a look again, I hope I explained everything and made the required adjustments.

gadididi

looks good!
thanks :)

Madhu-1 · 2026-02-11T05:32:22Z

@Merigfyio queue

nixpanic · 2026-02-11T11:53:29Z

@Mergifyio queue

mergify · 2026-02-11T11:53:38Z

mergify Bot added the component/nvme-of Issues and PRs related to NVMe-oF. label Oct 9, 2025

nixpanic added component/testing Additional test cases or CI work ci/skip/multi-arch-build skip building on multiple architectures labels Oct 9, 2025

This was referenced Oct 9, 2025

ci: add nvmeof test-type support for triggering CI jobs #5645

Merged

nvmeof: adding of nvmeof csi driver #5633

Merged

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 4d5676f to 0298e90 Compare October 10, 2025 08:05

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 0298e90 to dd3d82d Compare October 10, 2025 08:49

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 19de06a to be0d918 Compare October 10, 2025 15:23

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 394858c to 77aacdb Compare October 12, 2025 12:27

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 77aacdb to 72e0e02 Compare October 13, 2025 06:57

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from 14a2ccf to 7af1a69 Compare October 13, 2025 10:56

gadididi reviewed Feb 9, 2026

View reviewed changes

Madhu-1 reviewed Feb 9, 2026

View reviewed changes

nixpanic commented Feb 10, 2026

View reviewed changes

gadididi and others added 12 commits February 10, 2026 11:22

doc: add examples for NVMe-oF

6d85503

Add example of storage class and pvc, need to modify storage class before use! Signed-off-by: Niels de Vos <ndevos@ibm.com> Signed-off-by: gadi-didi <gadi.didi@ibm.com>

deploy: add deployment files for NVMe-oF

7441e91

The deployment files make it possible to start the NVMe-oF controller and nodeplugin by the e2e suite. Co-authored-by: gadi-didi <gadi.didi@ibm.com> Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: deploy NVMe-oF provisioner and nodeplugin

73aea58

Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: create NVMe-oF StorageClass

671becc

Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: add NVMe-oF volume creation test

db292a0

Create a PVC and wait for it to get Bound. This does not exersice the node-plugin yet, only the controller/provisioner. Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: enable nvmeof testing by default

ecfd07f

Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: record the logs from the gateway Pod in case it fails to start

cebf17f

Signed-off-by: Niels de Vos <ndevos@ibm.com>

ci: use Rook 1.18.4

243a596

Recent versions of Rook use Ceph-CSI Operator by default. There is no need to have Ceph-CSI deployed by Rook, as our CI deploys it from the PR that is being tested. Signed-off-by: Niels de Vos <ndevos@ibm.com>

e2e: log details of the NVMe-oF deployment on failure

3d1a0fb

Signed-off-by: Niels de Vos <ndevos@ibm.com>

doc: add examples for VAC NVMe-oF

5b5278d

Add example of volumeAttributesClass yaml files. it is possible to use in with PVC, by using the field volumeAttributesClassName. Co-authored-by: Niels de Vos <ndevos@ibm.com> Signed-off-by: gadi-didi <gadi.didi@ibm.com>

e2e: skip NVMe-oF testing on Ceph clusters < 20.0

17d5ccc

Ceph Tentacle is required for deploying the NVMe-oF gateway. Earlier versions do not support the gateway. Signed-off-by: Niels de Vos <ndevos@ibm.com>

nixpanic force-pushed the nvmeof/e2e-pvc-bound branch from e0bb01c to 17d5ccc Compare February 10, 2026 10:22

nixpanic requested review from Madhu-1 and gadididi February 10, 2026 10:22

gadididi approved these changes Feb 10, 2026

View reviewed changes

Madhu-1 approved these changes Feb 11, 2026

View reviewed changes

mergify Bot added the queued label Feb 11, 2026

mergify Bot added a commit that referenced this pull request Feb 11, 2026

Merge of #5641

a689d1c

mergify Bot mentioned this pull request Feb 11, 2026

merge queue: embarking devel (824eb99) and #5641 together #6051

Closed

78 tasks

mergify Bot merged commit 8c16324 into ceph:devel Feb 11, 2026
17 checks passed

mergify Bot removed the queued label Feb 11, 2026

		# FIXME: delete imagePullPolicy, or set to "IfNotPresent"
		imagePullPolicy: "Always"


		# subsystemNQN is the NQN of the subsystem where all volumes belong to
		subsystemNQN: "nqn.2025-08.io.ceph:k8s-ceph-csi"

		validateRBDImageCount(f, 1, nvmeofPool)
		validateOmapCount(f, 1, rbdType, nvmeofPool, volumesType)

		@@ -0,0 +1,13 @@
		---
		# apiVersion: storage.k8s.io/v1beta1 for k8s version 1.33

Conversation

nixpanic commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nixpanic commented Oct 9, 2025

Uh oh!

nixpanic commented Oct 9, 2025

Uh oh!

nixpanic commented Oct 9, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 10, 2025

Uh oh!

nixpanic commented Oct 12, 2025

Uh oh!

nixpanic commented Oct 13, 2025

Uh oh!

nixpanic commented Oct 13, 2025

Uh oh!

nixpanic commented Oct 13, 2025

Uh oh!

gadididi left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Madhu-1 left a comment

nixpanic commented Oct 9, 2025 •

edited

Loading