Ceph » Orchestrator

Run : https://pulpito.ceph.com/saraut-2024-08-29_09:15:11-rados:dashboard-wip-saraut02-testing-2024-08-26-1653-distro-default-smithi/
Jobs : [7879889, 7879891]

Logs for one : https://qa-proxy.ceph.com/teuthology/saraut-2024-08-29_09:15:11-rados:dashboard-wip-saraut02-testing-2024-08-26-1653-distro-default-smithi/7879889/teuthology.log

Log snippet -

2024-08-30T02:38:28.265 DEBUG:teuthology.orchestra.run.smithi050:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-08-30T02:38:28.269 DEBUG:teuthology.orchestra.run.smithi062:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest
2024-08-30T02:38:28.280 INFO:teuthology.orchestra.run.smithi050.stdout:   262217      4 drwxr-xr-x   3 ubuntu   ubuntu       4096 Aug 30 02:38 /home/ubuntu/cephtest
2024-08-30T02:38:28.280 INFO:teuthology.orchestra.run.smithi050.stdout:   419850      4 drwxr-xr-x   2 ubuntu   ubuntu       4096 Aug 30 02:37 /home/ubuntu/cephtest/ceph.data
2024-08-30T02:38:28.280 INFO:teuthology.orchestra.run.smithi050.stdout:   284685      4 -rw-r--r--   1 ubuntu   ubuntu        201 Aug 30 02:37 /home/ubuntu/cephtest/ceph.monmap
2024-08-30T02:38:28.285 INFO:teuthology.orchestra.run.smithi050.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty
2024-08-30T02:38:28.287 DEBUG:teuthology.orchestra.run:got remote process result: 1
2024-08-30T02:38:28.287 ERROR:teuthology.run_tasks:Manager failed: internal.base
Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 48, in base
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19f44d0aaed494e78709c5252ec9b827fa1dad82/qa/tasks/cephadm.py", line 2302, in task
    with contextutil.nested(
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 54, in nested
    raise exc[1]
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19f44d0aaed494e78709c5252ec9b827fa1dad82/qa/tasks/cephadm.py", line 1930, in initialize_config
    yield
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/contextutil.py", line 46, in nested
    if exit(*exc):
  File "/usr/lib/python3.10/contextlib.py", line 142, in __exit__
    next(self.gen)
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19f44d0aaed494e78709c5252ec9b827fa1dad82/qa/tasks/cephadm.py", line 461, in ceph_log
    if first_in_ceph_log('\[ERR\]|\[WRN\]|\[SEC\]',
  File "/home/teuthworker/src/github.com_ceph_ceph-c_19f44d0aaed494e78709c5252ec9b827fa1dad82/qa/tasks/cephadm.py", line 445, in first_in_ceph_log
    r = ctx.ceph[cluster_name].bootstrap_remote.run(
AttributeError: 'Namespace' object has no attribute 'bootstrap_remote'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/run_tasks.py", line 160, in run_tasks
    suppress = manager.__exit__(*exc_info)
  File "/usr/lib/python3.10/contextlib.py", line 153, in __exit__
    self.gen.throw(typ, value, traceback)
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/task/internal/__init__.py", line 53, in base
    run.wait(
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 479, in wait
    proc.wait()
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 161, in wait
    self._raise_for_status()
  File "/home/teuthworker/src/git.ceph.com_teuthology_3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a/teuthology/orchestra/run.py", line 181, in _raise_for_status
    raise CommandFailedError(
teuthology.exceptions.CommandFailedError: Command failed on smithi050 with status 1: 'find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest'
2024-08-30T02:38:28.300 DEBUG:teuthology.run_tasks:Unwinding manager kernel
2024-08-30T02:38:28.310 DEBUG:teuthology.run_tasks:Unwinding manager console_log
2024-08-30T02:38:28.322 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: AttributeError: 'Namespace' object has no attribute 'bootstrap_remote'
2024-08-30T02:38:28.325 INFO:teuthology.run:Summary data:
description: rados:dashboard/{0-single-container-host debug/mgr mon_election/connectivity
  random-objectstore$/{bluestore-comp-snappy} tasks/e2e}
duration: 515.8482844829559
failure_reason: 'Command failed on smithi050 with status 1: ''yes | sudo mkfs.xfs
  -f -i size=2048 /dev/vg_nvme/lv_2'''
flavor: default
owner: scheduled_saraut@teuthology
sentry_event: https://sentry.ceph.com/organizations/ceph/?query=fedc150b883a45c58d16c125547bf037
status: fail
success: false