Skip to content

Add auto listeners with --network-mask#1381

Merged
VallariAg merged 10 commits intoceph:develfrom
VallariAg:default-listeners
Dec 17, 2025
Merged

Add auto listeners with --network-mask#1381
VallariAg merged 10 commits intoceph:develfrom
VallariAg:default-listeners

Conversation

@VallariAg
Copy link
Member

@VallariAg VallariAg commented Jun 26, 2025 •
edited
Loading

Fixes: https://tracker.ceph.com/issues/71860

ceph PR: ceph/ceph#64210

# create subsystem with `--default-listeners`
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.x.x --server-port 5500 subsystem add -n nqn.2016-06.io.spdk:cnode4.mygroup1  --no-group-append --default-listeners

# verify listeners are added to each gateway with "gw listener_info" command
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.x.x --server-port 5500 gw listener_info -n nqn.2016-06.io.spdk:cnode4.mygroup1
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.x.y --server-port 5500 gw listener_info -n nqn.2016-06.io.spdk:cnode4.mygroup1
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.x.z --server-port 5500 gw listener_info -n nqn.2016-06.io.spdk:cnode4.mygroup1
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.x.k --server-port 5500 gw listener_info -n nqn.2016-06.io.spdk:cnode4.mygroup1

# verify by connecting to one of the other gw
# 1. add ns by "x.x.x.k" gw
docker run --rm quay.io/vallari/nvmeof-cli:wip-default-listeners-debug-2 --server-address x.x.xk --server-port 5500 namespace add -n nqn.2016-06.io.spdk:cnode4.mygroup1  --rbd-pool mypool --rbd-image myimage4

# 2. connect to above ns on "x.x.x.k" gw
nvme connect -t tcp --traddr x.x.x.k -s 4420 -n nqn.2016-06.io.spdk:cnode4.mygroup1
nvme list
Node                  Generic               SN                   Model                                    Namespace  Usage                      Format           FW Rev
--------------------- --------------------- -------------------- ---------------------------------------- ---------- -------------------------- ---------------- --------
/dev/nvme0n1          /dev/ng0n1            Ceph6253472416666    Ceph bdev Controller                     0x1          8.59  GB /   8.59  GB    512   B +  0 B   24.09

@github-project-automation github-project-automation bot moved this to 🆕 New in NVMe-oF Jun 26, 2025
@VallariAg VallariAg force-pushed the default-listeners branch from a87d303 to de6e51c Compare June 26, 2025 13:58
@VallariAg VallariAg mentioned this pull request Jun 26, 2025
Closed
14 tasks
@gadididi gadididi mentioned this pull request Sep 1, 2025
Merged
6 tasks
@gadididi gadididi mentioned this pull request Sep 14, 2025
Merged
6 tasks
@VallariAg
Copy link
Member Author

VallariAg commented Oct 17, 2025

auto-listeners:

[root@initnvme-vm15 ~]# sudo docker run -it quay.io/vallari/nvmeof-cli:auto-listeners --server-address x.x.x.x --server-port 5500 subsystem add -n nqn.2016-06.io.spdk:cnode99 --network-mask "x.x.x.0/24"
Adding subsystem nqn.2016-06.io.spdk:cnode99.mygroup1: Successful
[root@initnvme-vm15 ~]#
[root@initnvme-vm15 ~]#
[root@initnvme-vm15 ~]# docker run --rm quay.io/vallari/nvmeof-cli:auto-listeners --server-address x.x.x.x --server-port 5500 listener list -n nqn.2016-06.io.spdk:cnode99.mygroup1
Listeners for nqn.2016-06.io.spdk:cnode99.mygroup1:
╒═══════════════╤═════════════╤══════════════════╤═══════════════════╤══════════╤══════════╕
│ Host          │ Transport   │ Address Family   │ Address           │ Secure   │ Active   │
╞═══════════════╪═════════════╪══════════════════╪═══════════════════╪══════════╪══════════╡
│ cephnvme-vm11 │ TCP         │ IPv4             │ xx.xx.xx.a:4420 │ No       │ No       │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm12 │ TCP         │ IPv4             │ xx.xx.xx.b:4420 │ No       │ No       │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm13 │ TCP         │ IPv4             │ xx.xx.xx.c:4420 │ No       │ Yes      │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm14 │ TCP         │ IPv4             │ xx.xx.xx.d:4420 │ No       │ No       │
╘═══════════════╧═════════════╧══════════════════╧═══════════════════╧══════════╧══════════╛
[root@initnvme-vm15 ~]#

[root@initnvme-vm15 ~]# docker run --rm  quay.io/vallari/nvmeof-cli:auto-listeners --server-address 10.242.64.37 --server-port 5500 host add --subsystem nqn.2016-06.io.spdk:cnode99.mygroup1 --host-nqn "*"
Allowing open host access to nqn.2016-06.io.spdk:cnode99.mygroup1: Successful
Open host access to subsystem nqn.2016-06.io.spdk:cnode99.mygroup1 might be a security breach
[root@initnvme-vm15 ~]#


[root@initnvme-vm15 ~]# nvme discover -t tcp -a 10.242.64.36 -s 8009
...
...
...
=====Discovery Log Entry 20======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  20
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode99.mygroup1
traddr:  x.x.x.a
eflags:  none
sectype: none
=====Discovery Log Entry 21======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  21
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode99.mygroup1
traddr:  x.x.x.b
eflags:  none
sectype: none
=====Discovery Log Entry 22======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  22
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode99.mygroup1
traddr:  x.x.x.c
eflags:  none
sectype: none
=====Discovery Log Entry 23======
trtype:  tcp
adrfam:  ipv4
subtype: nvme subsystem
treq:    not required
portid:  23
trsvcid: 4420
subnqn:  nqn.2016-06.io.spdk:cnode99.mygroup1
traddr:  x.x.x.d
eflags:  none
sectype: none

secure listeners:

[root@initnvme-vm15 ~]# sudo docker run -it quay.io/vallari/nvmeof-cli:auto-listeners --server-address x.x.x.a --server-port 5500 subsystem add -n nqn.2016-06.io.spdk:cnode98 --network-mask "x.x.x.0/24" --secure-listeners
Adding subsystem nqn.2016-06.io.spdk:cnode98.mygroup1: Successful
[root@initnvme-vm15 ~]#
[root@initnvme-vm15 ~]#
[root@initnvme-vm15 ~]# docker run --rm quay.io/vallari/nvmeof-cli:auto-listeners --server-address x.x.x.a --server-port 5500 listener list -n nqn.2016-06.io.spdk:cnode98.mygroup1
Listeners for nqn.2016-06.io.spdk:cnode98.mygroup1:
╒═══════════════╤═════════════╤══════════════════╤═══════════════════╤══════════╤══════════╕
│ Host          │ Transport   │ Address Family   │ Address           │ Secure   │ Active   │
╞═══════════════╪═════════════╪══════════════════╪═══════════════════╪══════════╪══════════╡
│ cephnvme-vm11 │ TCP         │ IPv4             │ x.x.x.a:4420 │ Yes      │ No       │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm12 │ TCP         │ IPv4             │ x.x.x.b:4420 │ Yes      │ No       │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm13 │ TCP         │ IPv4             │ x.x.x.c:4420 │ Yes      │ Yes      │
├───────────────┼─────────────┼──────────────────┼───────────────────┼──────────┼──────────┤
│ cephnvme-vm14 │ TCP         │ IPv4             │ x.x.x.d:4420 │ Yes      │ No       │
╘═══════════════╧═════════════╧══════════════════╧═══════════════════╧══════════╧══════════╛
[root@initnvme-vm15 ~]#

During update, auto-listeners are created after normal listeners are created, preserving the order of subsystem -> host -> namespaces -> listeners.

@VallariAg VallariAg changed the title Add auto listeners with --default-listeners Add auto listeners with --network-mask Nov 5, 2025
@VallariAg VallariAg force-pushed the default-listeners branch 2 times, most recently from c83f863 to c5d1004 Compare November 17, 2025 06:50
@VallariAg VallariAg mentioned this pull request Nov 19, 2025
Merged
14 tasks
@VallariAg VallariAg requested a review from gbregman November 19, 2025 05:46
gbregman
gbregman requested changes Nov 19, 2025
View reviewed changes
Copy link
Collaborator

@gbregman gbregman left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems OK but there are some minor comments.

@github-project-automation github-project-automation bot moved this from 🆕 New to ♻ Changes requested in NVMe-oF Nov 19, 2025
@VallariAg VallariAg force-pushed the default-listeners branch 2 times, most recently from c362b16 to 180335a Compare December 8, 2025 07:47
@VallariAg VallariAg requested a review from gbregman December 8, 2025 07:48
gbregman
gbregman requested changes Dec 8, 2025
View reviewed changes
@VallariAg VallariAg force-pushed the default-listeners branch 3 times, most recently from 473a53c to af03176 Compare December 15, 2025 08:46
@gbregman gbregman self-requested a review December 15, 2025 09:23
@github-project-automation github-project-automation bot moved this from ♻ Changes requested to 👍 Ready to merge in NVMe-oF Dec 15, 2025
@VallariAg
cephutils: Add get_gw_listeners method
00825aa 
Return result of new 'nvme-gw listeners' mon command.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
utils: Add NICS.get_ips_in_subnet() method
57c6f23 
This method picks IP address from host which
belong to 'subnet' which would be in CIDR notation.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
Add --network-mask and --secure-listeners in 'subsystem add'
1a11efd 
Optional '--network-mask' param accepts CIDR subnet input.
This automatically add listeners for that subsystem on each
host, by picking IPs within the given subnet.

Optional boolean '--secure-listeners' param means that
all auto-listeners created would be secure listeners.

This commit also ensures "listeners list" and "nvme discover"
also display these auto-listeners.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
discovery: Use omap if "nvme-gw listeners" cmd fails
bb72f5c 
Show listeners form OMAP, if "nvme-gw listeners"
cmd fails. To add backport compatibility.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
Add GatewayUtils.get_hostname method
1022d4e 
And fix host_name for auto-listeners in
"list_listeners" grpc method.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
During update, add auto-listeners after namespaces
18fb1cd 
During updates, ensure the  auto-listeners are created
after namespaces like normal listeners.

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
Show if listener is auto-listener or custom-made
afd1081 
Also in gateway.proto, add 'custom' to listener_info

Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg VallariAg force-pushed the default-listeners branch 2 times, most recently from 9f3f4b4 to 4569bb8 Compare December 16, 2025 09:32
@VallariAg
CI: Add tests/ha/auto_listeners.sh
7149340 
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
Add auto-listener pytest
7b5e311 
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg
Minor fixes on auto-listeners
6e9dd5f 
Signed-off-by: Vallari Agrawal <val.agl002@gmail.com>
@VallariAg VallariAg merged commit 3b22ca2 into ceph:devel Dec 17, 2025
165 of 167 checks passed
@github-project-automation github-project-automation bot moved this from 👍 Ready to merge to ✅ Done in NVMe-oF Dec 17, 2025
This was referenced Dec 29, 2025
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gbregman gbregman gbregman approved these changes

Assignees

No one assigned

Labels

None yet

Projects

NVMe-oF
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@VallariAg @gbregman