Ceph » CephFS

Failed job:¶

/a/yuriw-2026-03-04_22:27:22-smoke-wip-rocky10-branch-of-the-day-2026-03-04-1772633736-distro-default-trial/83978

2026-03-05T12:47:39.803 INFO:teuthology.orchestra.run.trial160.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0
2026-03-05T12:47:39.803 INFO:teuthology.orchestra.run.trial160.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0
2026-03-05T12:47:39.804 INFO:teuthology.orchestra.run:Running command with timeout 300
2026-03-05T12:47:39.804 DEBUG:teuthology.orchestra.run.trial160:> ls /sys/fs/fuse/connections
2026-03-05T12:47:40.952 DEBUG:teuthology.orchestra.run.trial160:> sudo modprobe fuse
2026-03-05T12:47:40.962 DEBUG:teuthology.orchestra.run.trial160:> cat /proc/self/mounts | awk '{print $2}'
...
2026-03-05T12:48:17.104 WARNING:tasks.cephfs.fuse_mount:Trying to clean up after failed mount
2026-03-05T12:48:17.105 DEBUG:teuthology.orchestra.run.trial160:> set -ex
2026-03-05T12:48:17.105 DEBUG:teuthology.orchestra.run.trial160:> dd if=/proc/self/mounts of=/dev/stdout
2026-03-05T12:48:17.109 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is:
sysfs /sys sysfs rw,nosuid,nodev,noexec,relatime 0 0
proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0
udev /dev devtmpfs rw,nosuid,relatime,size=65421872k,nr_inodes=16355468,mode=755,inode64 0 0
devpts /dev/pts devpts rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0
tmpfs /run tmpfs rw,nosuid,nodev,noexec,relatime,size=13096172k,mode=755,inode64 0 0
/dev/nvme0n1p2 / ext4 rw,relatime 0 0
securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0
tmpfs /dev/shm tmpfs rw,nosuid,nodev,inode64 0 0
tmpfs /run/lock tmpfs rw,nosuid,nodev,noexec,relatime,size=5120k,inode64 0 0
cgroup2 /sys/fs/cgroup cgroup2 rw,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0
pstore /sys/fs/pstore pstore rw,nosuid,nodev,noexec,relatime 0 0
efivarfs /sys/firmware/efi/efivars efivarfs rw,nosuid,nodev,noexec,relatime 0 0
bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0
systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=17394 0 0
hugetlbfs /dev/hugepages hugetlbfs rw,relatime,pagesize=2M 0 0
mqueue /dev/mqueue mqueue rw,nosuid,nodev,noexec,relatime 0 0
debugfs /sys/kernel/debug debugfs rw,nosuid,nodev,noexec,relatime 0 0
tracefs /sys/kernel/tracing tracefs rw,nosuid,nodev,noexec,relatime 0 0
nfsd /proc/fs/nfsd nfsd rw,relatime 0 0
fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0
configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0
none /run/credentials/systemd-sysusers.service ramfs ro,nosuid,nodev,noexec,relatime,mode=700 0 0
/dev/loop0 /snap/core20/2686 squashfs ro,nodev,relatime,errors=continue 0 0
/dev/loop1 /snap/lxd/36918 squashfs ro,nodev,relatime,errors=continue 0 0
/dev/loop2 /snap/snapd/25935 squashfs ro,nodev,relatime,errors=continue 0 0
/dev/nvme0n1p1 /boot/efi vfat rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,errors=remount-ro 0 0
binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0
sunrpc /run/rpc_pipefs rpc_pipefs rw,relatime 0 0
tmpfs /run/snapd/ns tmpfs rw,nosuid,nodev,noexec,relatime,size=13096172k,mode=755,inode64 0 0
nsfs /run/snapd/ns/lxd.mnt nsfs rw 0 0
tmpfs /run/user/1000 tmpfs rw,nosuid,nodev,relatime,size=13096168k,nr_inodes=3274042,mode=700,uid=1000,gid=1002,inode64 0 0
/dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0
tmpfs /run/netns tmpfs rw,nosuid,nodev,noexec,relatime,size=13096172k,mode=755,inode64 0 0
nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0
nsfs /run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 nsfs rw 0 0

2026-03-05T12:48:17.109 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse client.0 is not mounted at ubuntu@trial160.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0
2026-03-05T12:48:17.109 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@trial160.front.sepia.ceph.com
2026-03-05T12:48:17.109 INFO:teuthology.orchestra.run:Running command with timeout 300
2026-03-05T12:48:17.110 DEBUG:teuthology.orchestra.run.trial160:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0)
2026-03-05T12:48:17.157 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_6e3d0419ccca0e75a03e08feb708946e198236f4/teuthology/run_tasks.py", line 112, in run_tasks
    manager.__enter__() 
  File "/usr/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3b19d231df1ddad45c9e95b328a8c12776aa5d/qa/tasks/ceph_fuse.py", line 234, in task
    mount_x.mount(mntopts=config.get('mntopts', []), mntargs=mntargs)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3b19d231df1ddad45c9e95b328a8c12776aa5d/qa/tasks/cephfs/fuse_mount.py", line 52, in mount
    return self._mount(mntopts, mntargs, check_status)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3b19d231df1ddad45c9e95b328a8c12776aa5d/qa/tasks/cephfs/fuse_mount.py", line 68, in _mount
    retval = self._run_mount_cmd(mntopts, mntargs, check_status)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3b19d231df1ddad45c9e95b328a8c12776aa5d/qa/tasks/cephfs/fuse_mount.py", line 94, in _run_mount_cmd
    return self._wait_and_record_our_fuse_conn(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/teuthworker/src/github.com_ceph_ceph-c_ed3b19d231df1ddad45c9e95b328a8c12776aa5d/qa/tasks/cephfs/fuse_mount.py", line 189, in _wait_and_record_our_fuse_conn
    raise RuntimeError( 
RuntimeError: Fuse mount failed to populate/sys/ after 31 seconds 

Several more failed in the same run:
3 jobs: ['83978', '83979', '83982']

Passed job (exact same job description):¶

zack-2026-02-04_16:34:20-smoke-main-distro-default-trial/35060

2026-02-04T19:12:05.560 INFO:teuthology.orchestra.run.trial095.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0
2026-02-04T19:12:05.560 INFO:teuthology.orchestra.run.trial095.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0
2026-02-04T19:12:05.561 INFO:teuthology.orchestra.run:Running command with timeout 300
2026-02-04T19:12:05.561 DEBUG:teuthology.orchestra.run.trial095:> ls /sys/fs/fuse/connections
2026-02-04T19:12:05.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.trial095.stderr:2026-02-04T19:12:05.573+0000 7fb77d837680 -1 init, newargv = 0x557483dd1ab0 newargc=15
2026-02-04T19:12:05.575 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.trial095.stderr:ceph-fuse[11681]: starting ceph client
2026-02-04T19:12:05.580 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.trial095.stderr:ceph-fuse[11681]: starting fuse
2026-02-04T19:12:06.680 DEBUG:teuthology.orchestra.run.trial095:> sudo modprobe fuse
2026-02-04T19:12:06.689 DEBUG:teuthology.orchestra.run.trial095:> cat /proc/self/mounts | awk '{print $2}'

Almost certain that this is not related to Rocky10 or the related Python changes, but given that this issue was not previously tracked anywhere, and I can't find it on any other main baselines, we should rule out any relation.