Ceph » CephFS

@Rishabh Dave Let's get an update/RCA on this soon.

test_single_path_authorize_on_nonalphanumeric_fsname has 2 clients -- self.mount_a and self.mount_b. At the beginning of this test former client is umounted, CephFS already available on the cluster is deleted, a new FS is created, self.mount_a mounts this new FS and then rest of testing proceed on it. During teardown, attempts to unmount both is made. Former is unmounted smoothly but for latter client umounting hangs and causes test to crash.

To prevent this there are 2 solutions: unmount self.mount_b at the beginnig of test and keep only 1 client throughout the test or remount self.mount_b to the new FS. Former is better since only one client is required throughout the test second client is redundant. Code for it is posted here - https://github.com/ceph/ceph/pull/58311.

However, this patch doesn't seem to be the right approach to fix this. Since without the patch the test passes with FUSE client(1) and fails with kernel client(2), the bug lies in kclient rather than this test code.

From (1) -

The issue is same as what was seen originally -

2024-06-27T17:14:38.903 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.1...
2024-06-27T17:14:38.903 INFO:teuthology.orchestra.run:Running command with timeout 300
2024-06-27T17:14:38.903 DEBUG:teuthology.orchestra.run.smithi159:> sudo umount /home/ubuntu/cephtest/mnt.1
2024-06-27T17:14:55.246 DEBUG:teuthology.orchestra.run.smithi067:> sudo logrotate /etc/logrotate.d/ceph-test.conf
...(removing repetitive entries logrotate command)
2024-06-27T17:19:26.317 DEBUG:teuthology.orchestra.run.smithi159:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-06-27T17:19:38.907 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr
    out = self.in_stderr_buffer.read(nbytes, self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read
    raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

and it was reproducible with vstart_runner originally but it isn't anymore. Perhaps because kernel has been upgraded since then.

@Venky Shankar
cc @Xiubo Li

(1) https://pulpito.ceph.com/rishabh-2024-06-27_16:29:08-fs:functional-main-testing-default-smithi/7776315
(2) https://pulpito.ceph.com/rishabh-2024-06-27_16:28:49-fs:functional-main-testing-default-smithi/7776312

The patch at https://github.com/ceph/ceph/pull/58311 works fine -

kernel: https://pulpito.ceph.com/rishabh-2024-06-27_16:29:14-fs:functional-main-testing-default-smithi/7776318/teuthology.log
fuse: http://qa-proxy.ceph.com/teuthology/rishabh-2024-06-27_16:29:26-fs:functional-main-testing-default-smithi/7776321/teuthology.log

FUSE job failed but much much after test_single_path_authorize_on_nonalphanumeric_fsname passes.

the bug lies in kclient rather than this test code.

If this is correct, I propose that we add a separate test case that reproduces this kclient bug. And, in the mean time, we remove the unusued client (at least temporarily) from this test, so that test can be run as usual when PRs are tested against main branch. This ensures that the test coverage is not reduced while kclient is being fixed

@Xiubo Li I think this bug is when the stock kernel gets chosen for the run: https://pulpito.ceph.com/rishabh-2024-06-27_16:28:49-fs:functional-main-testing-default-smithi/7776312/

@Rishabh Dave mentioned that you were unable to reproduce this with the testing kernel.

Venky Shankar wrote in #note-8:

@Xiubo Li I think this bug is when the stock kernel gets chosen for the run: https://pulpito.ceph.com/rishabh-2024-06-27_16:28:49-fs:functional-main-testing-default-smithi/7776312/

@Rishabh Dave mentioned that you were unable to reproduce this with the testing kernel.

Xiubo didn't try to reproduce this (AFAIK). He checked if the CephFS, which was mounted using testing kernel, can be unmounted after MDS has been killed without hanging or crashingo or using --force or --lazy.

Rishabh Dave wrote in #note-9:

Venky Shankar wrote in #note-8:

@Xiubo Li I think this bug is when the stock kernel gets chosen for the run: https://pulpito.ceph.com/rishabh-2024-06-27_16:28:49-fs:functional-main-testing-default-smithi/7776312/

@Rishabh Dave mentioned that you were unable to reproduce this with the testing kernel.

Xiubo didn't try to reproduce this (AFAIK). He checked if the CephFS, which was mounted using testing kernel, can be unmounted after MDS has been killed without hanging or crashingo or using --force or --lazy.

OK, I misunderstood our conversation then. When the MDS is not reachable, the user should pass --force to allow unmounting otherwise umount would hang due to the client trying to flush data to the MDS.

So, it's pretty obvious now that the client that's left unmounted initially after the file system is deleted is the one blocked on umount. So, either run the test with one client or umount the extra client. Would that suffice, @Rishabh Dave ?

So, either run the test with one client or umount the extra client. Would that suffice, @Rishabh Dave ?

Yes. I've already implemented this approach and tested this. See: https://tracker.ceph.com/issues/66077#note-6