Ceph » Orchestrator

The nvme_loop task will at some point, after some initial setup, run something like `sudo nvme connect -t loop -n lv_1 -q hostnqn`. Currently, this command fails on tests using centos 9 on the host OS with

[root@smithi190 ~]# sudo nvme connect -t loop -n lv_1 -q hostnqn
Failed to write to /dev/nvme-fabrics: Invalid argument

dmesg reports something like

[ 1985.306386] nvme_fabrics: found same hostid edb4e426-766f-44c6-b127-da2a5b7446ef but different hostnqn hostnqn

and it can also be seen that the nvme-cli package on the host is unable to generate a hostnqn

[root@smithi190 ~]# echo $(nvme gen-hostnqn) > /etc/nvme/hostnqn
"gen-hostnqn" not supported. Install lib uuid and rebuild.

Uninstalling and reinstalling the nvme-cli package seems to fix the issue. It's worth noting the nvme-cli package relies on the libnvme package in order to generate the hostnqn, and that it does this generation during its postinstall scriplet

[root@smithi190 ~]# rpm -qlp nvme-cli-2.9.1-5.el9.x86_64.rpm --scripts
postinstall scriptlet (using /bin/sh):
if [ $1 -eq 1 ] || [ $1 -eq 2 ]; then
        if [ ! -s /etc/nvme/hostnqn ]; then
            echo $(nvme gen-hostnqn) > /etc/nvme/hostnqn
        fi
        if [ ! -s /etc/nvme/hostid ]; then
            echo $(nvme show-hostnqn | sed 's/^.*uuid://') > /etc/nvme/hostid
        fi

    # apply udev and systemd changes that we did
    if [ $1 -eq 1 ]; then
        systemctl enable nvmefc-boot-connections
    fi
    systemctl daemon-reload
    udevadm control --reload-rules && udevadm trigger
    exit 0
fi

The current theory is possiblly some sort of linking issue between the nvme-cli and libnvme package that fixes itself when the package is uninstalled and then installed again (which also triggers a reinstall of libnvme). Reinstalling libnvme on its own had no effect.