Ceph » CephFS

When an empty name is given while creating a subvolume, this will turn the subvolumegroup into a subvolume instead of giving a proper error or warning.

Example below uses a subvolumegroup argument for clarity (note that if omitted, the default "_nogroup" subbvolumegroup will be used).

ceph fs subvolumegroup create ceph-filesystem examplegroup
ceph fs subvolume create ceph-filesystem example1 --group_name examplegroup
ceph fs subvolume create ceph-filesystem "" --group_name examplegroup  # note the empty name
ceph fs subvolume create ceph-filesystem example2 --group_name examplegroup

This last command shows the following error:

Error EINVAL: invalid value specified for ceph.dir.subvolume

The error is correct, but not really helpful:
- the "invalid value specified" seems to be meant for the subvolumegroup, not the subvolume we try to create, which is confusing and misleading
- it is unclear which value is invalid, in this case a 1 needs to be a 0, but it requires thorough investigation to come to this conclusion

And to make it even more confusing; this extended attribute can't be read with getfattr, although it is used by Ceph:

# getfattr -n ceph.dir.subvolume /mnt/cephfs/volumes/examplegroup
/mnt/cephfs/volumes/examplegroup: ceph.dir.subvolume: No such attribute

Eventhough this extended attribute can't be read, it can be set with setfattr, to resolve the issue you need to change the "ceph.dir.subvolume" attribute on the filesystem to "0"

mount -t ceph 10.10.10.100:6789,10.10.10.101:6789,10.10.10.102:6789:/ /mnt/cephfs/ -o name=admin,secret=admin_secret
setfattr -n ceph.dir.subvolume -v 0 /mnt/cephfs/volumes/examplegroup

Tested on versions:
- quincy (17) shows error only as shown in example
- octopus (15) and pacific (16) show multiple stacktraces, result (unusable subbvolumegroup) is the same.

Stacktrace octopus|pacific:

Error EINVAL: Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 271, in discover
    self.fs.stat(self.base_path)
  File "cephfs.pyx", line 1338, in cephfs.LibCephFS.stat
cephfs.ObjectNotFound: error in stat: /volumes/examplegroup/example2: No such file or directory [Errno 2]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 164, in create_subvolume
    with open_subvol(self.mgr, fs_handle, self.volspec, group, subvolname, SubvolumeOpType.CREATE) as subvolume:
  File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
    return next(self.gen)
  File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 72, in open_subvol
    subvolume = loaded_subvolumes.get_subvolume_object(mgr, fs, vol_spec, group, subvolname)
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/__init__.py", line 95, in get_subvolume_object
    subvolume.discover()
  File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 283, in discover
    raise VolumeException(-errno.ENOENT, "subvolume '{0}' does not exist".format(self.subvolname))
volumes.fs.exception.VolumeException: -2 (subvolume 'example2' does not exist)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 101, in mark_subvolume
        self.fs.setxattr(self.base_path, 'ceph.dir.subvolume', b'1', 0)
      File "cephfs.pyx", line 1264, in cephfs.LibCephFS.setxattr
    cephfs.InvalidValue: error in setxattr: Invalid argument [Errno 22]

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 171, in create
        self.mark_subvolume()
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 103, in mark_subvolume
        raise VolumeException(-errno.EINVAL, "invalid value specified for ceph.dir.subvolume")
    volumes.fs.exception.VolumeException: -22 (invalid value specified for ceph.dir.subvolume)

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 99, in dump
        self.fs.rename(path, self.unique_trash_path)
      File "cephfs.pyx", line 1545, in cephfs.LibCephFS.rename
    cephfs.Error: error in rename /volumes/examplegroup/example2 to /volumes/_deleting/4dd2069d-7cf9-44fe-b1c9-1c3c8e80ae28: Invalid cross-device link [Errno 18]

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/share/ceph/mgr/mgr_module.py", line 1212, in _handle_command
        return self.handle_command(inbuf, cmd)
      File "/usr/share/ceph/mgr/volumes/module.py", line 466, in handle_command
        return handler(inbuf, cmd)
      File "/usr/share/ceph/mgr/volumes/module.py", line 34, in wrap
        return f(self, inbuf, cmd)
      File "/usr/share/ceph/mgr/volumes/module.py", line 520, in _cmd_fs_subvolume_create
        namespace_isolated=cmd.get('namespace_isolated', False))
      File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 176, in create_subvolume
        self._create_subvolume(fs_handle, volname, group, subvolname, **kwargs)
      File "/usr/share/ceph/mgr/volumes/fs/volume.py", line 142, in _create_subvolume
        self.mgr, fs_handle, self.volspec, group, subvolname, size, isolate_nspace, pool, oct_mode, uid, gid)
      File "/usr/share/ceph/mgr/volumes/fs/operations/subvolume.py", line 26, in create_subvol
        subvolume.create(size, isolate_nspace, pool, mode, uid, gid)
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 193, in create
        self._remove_on_failure(subvol_path, retained)
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 151, in _remove_on_failure
        self.remove()
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_v2.py", line 346, in remove
        self.trash_base_dir()
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 301, in trash_base_dir
        self._trash_dir(self.base_path)
      File "/usr/share/ceph/mgr/volumes/fs/operations/versions/subvolume_base.py", line 289, in _trash_dir
        trashcan.dump(path)
      File "/usr/share/ceph/mgr/volumes/fs/operations/trash.py", line 101, in dump
        raise VolumeException(-e.args[0], e.args[1])
    TypeError: bad operand type for unary -: 'str'