Actions
Bug #68568
opencephfs/qa : Command failed (workunit test fs/misc/rstats.sh) on smithi136 with status 1
% Done:
0%
Source:
Q/A
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Component(FS):
Labels (FS):
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Description
2024-10-15T11:52:05.568 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 145, in task
_spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 434, in _run_tests
remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/rstats.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e07487c6fc88daf5bf2df835d473f72d50e621d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh'
2024-10-15T11:52:09.332 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=04c16425fd54416882e7c56690c7941a
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/run_tasks.py", line 105, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/run_tasks.py", line 83, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 145, in task
_spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'),
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 293, in _spawn_on_all_clients
with parallel() as p:
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/github.com_ceph_ceph-c_e07487c6fc88daf5bf2df835d473f72d50e621d4/qa/tasks/workunit.py", line 434, in _run_tests
remote.run(
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/remote.py", line 523, in run
r = self._runner(client=self.ssh, name=self.shortname, **kwargs)
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 455, in run
r.wait()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_teuthology_e2eb3a97c43341ca720d42f59fa1648ddab2267a/teuthology/orchestra/run.py", line 181, in _raise_for_status
raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed (workunit test fs/misc/rstats.sh) on smithi136 with status 1: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=e07487c6fc88daf5bf2df835d473f72d50e621d4 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/fs/misc/rstats.sh'
2024-10-15T11:52:09.335 DEBUG:teuthology.run_tasks:Unwinding manager background_exec
2024-10-15T11:52:09.345 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi136.front.sepia.ceph.com
Updated by Venky Shankar over 1 year ago
- Assignee set to Kotresh Hiremath Ravishankar
- Target version set to v20.0.0
The actual failure is:
2024-10-15T11:52:03.432 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait=29 2024-10-15T11:52:03.432 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 29 -lt 30 ']' 2024-10-15T11:52:03.432 INFO:tasks.workunit.client.0.smithi136.stderr:++ getfattr --only-value -n ceph.dir.rsubdirs . 2024-10-15T11:52:03.438 INFO:tasks.workunit.client.0.smithi136.stderr:+ new_value=3 2024-10-15T11:52:03.438 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 3 == 3 ']' 2024-10-15T11:52:03.438 INFO:tasks.workunit.client.0.smithi136.stderr:+ sleep 1 2024-10-15T11:52:04.440 INFO:tasks.workunit.client.0.smithi136.stderr:+ wait=30 2024-10-15T11:52:04.440 INFO:tasks.workunit.client.0.smithi136.stderr:+ '[' 30 -lt 30 ']' 2024-10-15T11:52:04.440 INFO:tasks.workunit.client.0.smithi136.stderr:+ return 1 2024-10-15T11:52:04.441 DEBUG:teuthology.orchestra.run:got remote process result: 1
The check expects rsubdirs count to increase, but the checks fails until timeout.
@Kotresh Hiremath Ravishankar Handing this over to you since you have dealt with recursive stats in the MDS :)
Actions