Project

General

Profile

Actions

Bug #66946

closed

qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fs

Added by Xiubo Li over 1 year ago. Updated 5 months ago.

Status:
Resolved
Priority:
Normal
Assignee:
Category:
Testing
Target version:
% Done:

0%

Source:
Q/A
Backport:
quincy,reef,squid
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
qa, qa-failure
Pull request ID:
Tags (freeform):
backport_processed
Fixed In:
v19.3.0-4144-g052cb8deb1
Released In:
v20.2.0~2263
Upkeep Timestamp:
2025-11-01T01:35:56+00:00

Description

https://pulpito.ceph.com/jcollin-2024-07-01_10:56:30-fs-wip-jcollin-testing-20240701.061036-squid-distro-default-smithi/7781111/

2024-07-02T01:54:02.230 INFO:tasks.cephfs_test_runner:Starting test: test_fs_rename_fails_for_non_existent_fs (tasks.cephfs.test_admin.TestRenameCommand)
That renaming a non-existent file system fails.
2024-07-02T01:54:02.230 INFO:tasks.cephfs_test_runner:test_fs_rename_fails_for_non_existent_fs (tasks.cephfs.test_admin.TestRenameCommand)
...
2024-07-02T01:54:28.958 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:28.971+0000 7fb15f7fe640  1 -- 172.21.15.135:0/850310939 <== mon.0 v2:172.21.15.135:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0  v0) ==== 72+0+185586 (secure 0 0 0) 0x7fb15807a2d0 con 0x7fb16806ef40
2024-07-02T01:54:29.165 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:29.182+0000 7fb16c909640  1 -- 172.21.15.135:0/850310939 --> [v2:172.21.15.135:3300/0,v1:172.21.15.135:6789/0] -- mon_command({"prefix": "fs rename", "fs_name": "non_existent_fs", "new_fs_name": "new_fs", "yes_i_really_mean_it": true} v 0) -- 0x7fb168072e10 con 0x7fb16806ef40
2024-07-02T01:54:29.166 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:29.182+0000 7fb15f7fe640  1 -- 172.21.15.135:0/850310939 <== mon.0 v2:172.21.15.135:3300/0 7 ==== mon_command_ack([{"prefix": "fs rename", "fs_name": "non_existent_fs", "new_fs_name": "new_fs", "yes_i_really_mean_it": true}]=-2 File system 'non_existent_fs' does not exist v890) ==== 186+0+0 (secure 0 0 0) 0x7fb158079bd0 con 0x7fb16806ef40
2024-07-02T01:54:29.166 INFO:teuthology.orchestra.run.smithi135.stderr:Error ENOENT: File system 'non_existent_fs' does not exist
2024-07-02T01:54:29.170 DEBUG:teuthology.orchestra.run:got remote process result: 2
2024-07-02T01:54:29.171 INFO:teuthology.nuke:Clearing teuthology firewall rules...
2024-07-02T01:54:29.171 DEBUG:teuthology.orchestra.run.smithi135:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2024-07-02T01:54:29.211 DEBUG:teuthology.orchestra.run.smithi184:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore'
2024-07-02T01:54:29.271 INFO:teuthology.nuke:Cleared teuthology firewall rules.
2024-07-02T01:54:29.271 INFO:tasks.cephfs.mount:Terminating background process
2024-07-02T01:54:29.271 DEBUG:teuthology.orchestra.run.smithi135:> set -ex
...
2024-07-02T01:54:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:29.702+0000 7f7d059af640  1 -- 172.21.15.135:0/304998763 >> 172.21.15.135:0/304998763 conn(0x7f7d000d0ca0 msgr2=0x7f7d000b49d0 unknown :-1 s=STATE_NONE l=0).mark_down
2024-07-02T01:54:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:29.702+0000 7f7d059af640  1 -- 172.21.15.135:0/304998763 shutdown_connections
2024-07-02T01:54:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2024-07-02T01:54:29.702+0000 7f7d059af640  1 -- 172.21.15.135:0/304998763 wait complete.
2024-07-02T01:54:29.697 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0...
2024-07-02T01:54:29.697 INFO:teuthology.orchestra.run:Running command with timeout 300
2024-07-02T01:54:29.697 DEBUG:teuthology.orchestra.run.smithi135:> sudo umount /home/ubuntu/cephtest/mnt.0
2024-07-02T01:54:36.745 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:54:36.750 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:55:06.884 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:55:06.888 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:55:37.056 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:55:37.059 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:56:07.186 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:56:07.190 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:56:37.249 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:56:37.253 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:57:07.396 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
...
2024-07-02T01:58:37.682 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:59:07.824 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:59:07.829 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:59:29.701 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 745, in recv_stderr
    out = self.in_stderr_buffer.read(nbytes, self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read
    raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 323, in copy_file_to
    copy_to_log(src, logger, capture=stream, quiet=quiet)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 276, in copy_to_log
    for line in f:
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__
    line = self.readline()
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline
    new_data = self._read(n)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1374, in _read
    return self.channel.recv_stderr(size)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 747, in recv_stderr
    raise socket.timeout()
TimeoutError
2024-07-02T01:59:29.865 ERROR:teuthology:Uncaught exception (Hub)
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 697, in recv
    out = self.in_buffer.read(nbytes, self.timeout)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/buffered_pipe.py", line 154, in read
    raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 323, in copy_file_to
    copy_to_log(src, logger, capture=stream, quiet=quiet)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/teuthology/orchestra/run.py", line 276, in copy_to_log
    for line in f:
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 109, in __next__
    line = self.readline()
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/file.py", line 275, in readline
    new_data = self._read(n)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 1359, in _read
    return self.channel.recv(size)
  File "/home/teuthworker/src/git.ceph.com_teuthology_544fecbcd55f3d2b6f478823823ce40cbefef1d4/virtualenv/lib/python3.10/site-packages/paramiko/channel.py", line 699, in recv
    raise socket.timeout()
TimeoutError
2024-07-02T01:59:37.920 DEBUG:teuthology.orchestra.run.smithi135:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:59:37.927 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T01:59:59.982 INFO:tasks.ceph.mon.a.smithi135.stderr:2024-07-02T02:00:00.000+0000 7f31b65ce640 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 1 filesystem is degraded; 1 filesystem is offline; 9 pool(s) do not have an application enabled
...
2024-07-02T12:54:07.343 DEBUG:teuthology.orchestra.run.smithi184:> sudo logrotate /etc/logrotate.d/ceph-test.conf
2024-07-02T12:54:08.444 DEBUG:teuthology.exit:Got signal 15; running 1 handler...
2024-07-02T12:54:08.444 DEBUG:teuthology.task.console_log:Killing console logger for smithi135
2024-07-02T12:54:08.445 DEBUG:teuthology.task.console_log:Killing console logger for smithi184
2024-07-02T12:54:08.446 DEBUG:teuthology.exit:Finished running handlers                                                                        

Related issues 4 (0 open4 closed)

Related to CephFS - Bug #66077: qa/cephfs: unmount hangs after test_single_path_authorize_on_nonalphanumeric_fsname ResolvedRishabh Dave

Actions
Copied to CephFS - Backport #67432: squid: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fsResolvedXiubo LiActions
Copied to CephFS - Backport #67433: quincy: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fsRejectedXiubo LiActions
Copied to CephFS - Backport #67434: reef: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fsResolvedXiubo LiActions
Actions #1

Updated by Xiubo Li over 1 year ago

  • Status changed from New to Fix Under Review
  • Pull request ID set to 58583
Actions #2

Updated by Xiubo Li over 1 year ago

  • Related to Bug #66077: qa/cephfs: unmount hangs after test_single_path_authorize_on_nonalphanumeric_fsname added
Actions #4

Updated by Venky Shankar over 1 year ago

  • Category set to Testing
  • Status changed from Fix Under Review to Pending Backport
  • Target version set to v20.0.0
  • Source set to Q/A
  • Backport changed from quincy,squid to quincy,reef,squid
  • Labels (FS) qa, qa-failure added
Actions #5

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67432: squid: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fs added
Actions #6

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67433: quincy: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fs added
Actions #7

Updated by Upkeep Bot over 1 year ago

  • Copied to Backport #67434: reef: qa/cephfs: unmount hangs after test_fs_rename_fails_for_non_existent_fs added
Actions #8

Updated by Upkeep Bot over 1 year ago

  • Tags (freeform) set to backport_processed
Actions #9

Updated by Upkeep Bot 9 months ago

  • Status changed from Pending Backport to Resolved
  • Upkeep Timestamp set to 2025-07-08T18:24:25+00:00
Actions #10

Updated by Upkeep Bot 8 months ago

  • Merge Commit set to 052cb8deb1ea705537f245edba504efb3035dbec
  • Fixed In set to v19.3.0-4144-g052cb8deb1
  • Upkeep Timestamp changed from 2025-07-08T18:24:25+00:00 to 2025-08-02T04:50:34+00:00
Actions #11

Updated by Upkeep Bot 5 months ago

  • Released In set to v20.2.0~2263
  • Upkeep Timestamp changed from 2025-08-02T04:50:34+00:00 to 2025-11-01T01:35:56+00:00
Actions

Also available in: Atom PDF