Project

General

Profile

Actions

Bug #66088

closed

qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issues

Added by Rishabh Dave almost 2 years ago. Updated 5 months ago.

Status:
Resolved
Priority:
High
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Backport:
quincy,reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
qa-suite
Labels (FS):
qa, qa-failure
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v19.3.0-3065-gdbc9816d2e
Released In:
v20.2.0~2596
Upkeep Timestamp:
2025-11-01T01:32:31+00:00

Description

/a/rishabh-2024-05-16_20:21:40-fs:functional-main-testing-default-smithi/7709435/teuthology.log

2024-05-16T21:34:14.645 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2024-05-16T21:34:14.646 INFO:teuthology.orchestra.run:Running command with timeout 300
2024-05-16T21:34:14.646 DEBUG:teuthology.orchestra.run.smithi028:> sudo umount /home/ubuntu/cephtest/mnt.0
...
2024-05-16T21:39:13.186 DEBUG:teuthology.orchestra.run.smithi028:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-05-16T21:39:13.190 DEBUG:teuthology.orchestra.run.smithi174:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-05-16T21:39:14.649 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 745, in recv_stderr
    out = self.in_stderr_buffer.read(nbytes, self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read
    raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 323, in copy_file_to
    copy_to_log(src, logger, capture=stream, quiet=quiet)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/orchestra/run.py", line 276, in copy_to_log
    for line in f:
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__
    line = self.readline()
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline
    new_data = self._read(n)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1374, in _read
    return self.channel.recv_stderr(size)
  File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 747, in recv_stderr
    raise socket.timeout()
socket.timeout

Related issues 4 (0 open4 closed)

Related to CephFS - Bug #66160: qa/cephfs: allow max tests to run in test_admin.pyResolvedRishabh Dave

Actions
Copied to CephFS - Backport #66726: reef: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issuesResolvedRishabh DaveActions
Copied to CephFS - Backport #66727: quincy: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issuesRejectedVenky ShankarActions
Copied to CephFS - Backport #66728: squid: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issuesResolvedJos CollinActions
Actions #1

Updated by Venky Shankar almost 2 years ago

  • Status changed from New to Triaged
  • Assignee set to Rishabh Dave
  • Priority changed from Normal to High
  • Target version set to v20.0.0
  • Source set to Q/A
  • Backport set to quincy,reef,squid

No daemon logs available for debugging. How reproducible is this @Rishabh Dave ?

Actions #2

Updated by Rishabh Dave almost 2 years ago

  • Related to Bug #66160: qa/cephfs: allow max tests to run in test_admin.py added
Actions #3

Updated by Venky Shankar almost 2 years ago

Rishabh, the test does

    def test_fs_rename_fails_for_non_existent_fs(self):
        """ 
        That renaming a non-existent file system fails.
        """ 
        self.skipTest('This test is broken ATM; see '
                      'https://tracker.ceph.com/issues/66088')

        self.run_ceph_cmd(f'fs fail {self.fs.name}')
        self.run_ceph_cmd(f'fs set {self.fs.name} refuse_client_session true')
        sleep(5)
        try:
            self.run_ceph_cmd("fs rename non_existent_fs new_fs --yes-i-really-mean-it")
        except CommandFailedError as ce:
            self.assertEqual(ce.exitstatus, errno.ENOENT, "invalid error code on renaming a non-existent fs")
        else:
            self.fail("expected renaming of a non-existent file system to fail")

When the test ends, the clients would be unmounted, however, the the file system is offline, so unmounting would always fail. How did https://github.com/ceph/ceph/pull/53899 pass tests?

@Rishabh Dave

Actions #4

Updated by Venky Shankar almost 2 years ago

  • Category set to Testing
  • Status changed from Triaged to Fix Under Review
  • Assignee changed from Rishabh Dave to Venky Shankar
  • Pull request ID set to 57619
Actions #5

Updated by Rishabh Dave almost 2 years ago · Edited

@Venky Shankar Didn't know you too were working on this, I had a fix ready for this but was planning to post it along with rest of test_admin fixes for bugs we discovered last week.

Actions #6

Updated by Rishabh Dave almost 2 years ago

Posted a patch here (https://github.com/ceph/ceph/pull/57624) that I had written along with the fix for this issue.

Actions #7

Updated by Venky Shankar almost 2 years ago · Edited

Rishabh Dave wrote in #note-5:

@Venky Shankar Didn't know you too were working on this, I had a fix ready for this but was planning to post it along with rest of test_admin fixes for bugs we discovered last week.

I wasn't until this showed up in quincy squid and I debugged it and found the fix to be straight-forward.

Actions #8

Updated by Venky Shankar over 1 year ago

  • Status changed from Fix Under Review to Pending Backport
Actions #9

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #66726: reef: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issues added
Actions #10

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #66727: quincy: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issues added
Actions #11

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #66728: squid: qa/cephfs: test_fs_rename_fails_for_non_existent_fs fails due to unmount issues added
Actions #12

Updated by Upkeep Bot over 1 year ago

  • Tags (freeform) set to backport_processed
Actions #13

Updated by Upkeep Bot 9 months ago

  • Merge Commit set to dbc9816d2e22d0e130b5778bd0d9f3b5a08a2069
  • Fixed In set to v19.3.0-3065-gdbc9816d2e2
  • Upkeep Timestamp set to 2025-07-08T22:38:27+00:00
Actions #14

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-3065-gdbc9816d2e2 to v19.3.0-3065-gdbc9816d2e22
  • Upkeep Timestamp changed from 2025-07-08T22:38:27+00:00 to 2025-07-14T15:47:00+00:00
Actions #15

Updated by Upkeep Bot 8 months ago

  • Fixed In changed from v19.3.0-3065-gdbc9816d2e22 to v19.3.0-3065-gdbc9816d2e
  • Upkeep Timestamp changed from 2025-07-14T15:47:00+00:00 to 2025-07-14T21:38:01+00:00
Actions #16

Updated by Jos Collin 7 months ago

  • Status changed from Pending Backport to Resolved
Actions #17

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~2596
  • Upkeep Timestamp changed from 2025-07-14T21:38:01+00:00 to 2025-11-01T01:32:31+00:00
Actions

Also available in: Atom PDF