Actions
Bug #54992
closedpacific: rados/dashboard: tasks/dashboard: cannot stat '/etc/containers/registries.conf': No such file or directory
% Done:
0%
Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
v16.2.10-1595-g708b6e54a7c
Released In:
v16.2.11~16
Upkeep Timestamp:
2025-07-14T06:13:39+00:00
Tags:
Description
/a/yuriw-2022-03-17_14:54:32-rados-wip-yuri10-testing-2022-03-16-1432-pacific-distro-default-smithi/6742377
2022-03-17T17:06:57.876 DEBUG:teuthology.orchestra.run.smithi040:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-03-17T17:06:57.877 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi125.front.sepia.ceph.com
2022-03-17T17:06:57.877 DEBUG:teuthology.orchestra.run.smithi125:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-03-17T17:06:57.918 INFO:teuthology.orchestra.run.smithi125.stderr:cp: cannot stat '/etc/containers/registries.conf': No such file or directory
2022-03-17T17:06:57.919 DEBUG:teuthology.orchestra.run:got remote process result: 1
2022-03-17T17:06:57.920 ERROR:teuthology.run_tasks:Saw exception from tasks.
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
2022-03-17T17:06:58.017 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=9636d9d49b2b403ca15242337274d90f
Traceback (most recent call last):
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 91, in run_tasks
manager = run_one_task(taskname, ctx=ctx, config=config)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/run_tasks.py", line 70, in run_one_task
return task(**kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 149, in task
p.spawn(_exec_host, barrier, barrier_queue, remote[0], sudo, testdir, remote[1])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 84, in __exit__
for result in self:
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 98, in __next__
resurrect_traceback(result)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 30, in resurrect_traceback
raise exc.exc_info[1]
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/parallel.py", line 23, in capture_traceback
return func(*args, **kwargs)
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/task/pexec.py", line 60, in _exec_host
tor.wait([r])
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 479, in wait
proc.wait()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 161, in wait
self._raise_for_status()
File "/home/teuthworker/src/git.ceph.com_git_teuthology_a09009223dc44ca923b2e6ad4cd4c4fe88381b22/teuthology/orchestra/run.py", line 183, in _raise_for_status
node=self.hostname, label=self.label
teuthology.exceptions.CommandFailedError: Command failed on smithi125 with status 1: 'TESTDIR=/home/ubuntu/cephtest bash -s'
Updated by Kamoltat (Junior) Sirivadhna almost 4 years ago
/a/yuriw-2022-03-26_19:49:47-rados-wip-yuri10-testing-2022-03-22-1809-pacific-distro-default-smithi/6762737
Updated by Laura Flores almost 4 years ago
/a/yuriw-2022-05-04_20:09:21-rados-pacific-distro-default-smithi/6821407
Updated by Laura Flores almost 4 years ago
- Has duplicate Bug #54086: Permission denied when trying to unlink and open /var/log/ntpstats/... added
Updated by Laura Flores almost 4 years ago
/a/yuriw-2022-06-02_00:50:42-rados-wip-yuri4-testing-2022-06-01-1350-pacific-distro-default-smithi/6859624
Updated by Matan Breizman over 3 years ago
/a/yuriw-2022-08-22_21:19:34-rados-wip-yuri4-testing-2022-08-18-1020-pacific-distro-default-smithi/6986466
Updated by Laura Flores over 3 years ago
- Tags set to test-failure
- Priority changed from Normal to High
Marking this as high since it appears in the rados suite a lot.
/a/amathuri-2022-09-08_14:19:23-rados-wip-yuri5-testing-2022-09-06-1334-pacific-distro-default-smithi/7020554
Updated by Adam King over 3 years ago
- Project changed from Orchestrator to Dashboard
- Subject changed from cannot stat '/etc/containers/registries.conf': No such file or directory to pacific: rados/dashboard: tasks/dashboard: cannot stat '/etc/containers/registries.conf': No such file or directory
moving this to dashboard component given the failure only exists in a dashboard specific test. My guess is this is an issue with something the test is doing rather than an actual bug in Ceph or the dashboard, but maybe someone who knows the dashboard tests a bit better can investigate. It also seems to only happen in pacific if that info is useful.
Updated by Aishwarya Mathuria over 3 years ago
/a/yuriw-2022-11-29_22:29:58-rados-wip-yuri10-testing-2022-11-29-1005-pacific-distro-default-smithi/7097412
Updated by Laura Flores over 3 years ago
This bug occurs on Pacific whenever the test is run on ubuntu.
There are clear messages about ubuntu not supporting Selinux in the log:
2022-11-30T01:15:07.924 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}
2022-11-30T01:15:07.925 DEBUG:teuthology.task.selinux:Excluding smithi143: OS 'ubuntu' does not support SELinux
2022-11-30T01:15:07.925 DEBUG:teuthology.task.selinux:Excluding smithi175: OS 'ubuntu' does not support SELinux
Which leads to:
2022-11-30T01:19:48.277 INFO:teuthology.orchestra.run.smithi175.stdout:30 Nov 01:19:48 ntpd[17296]: Listening on routing socket on fd #22 for interface updates
2022-11-30T01:19:49.259 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:49.260 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:49.261 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2022-11-30T01:19:49.261 INFO:teuthology.orchestra.run.smithi143.stderr:30 Nov 01:19:49 ntpd[17328]: can't open /var/log/ntpstats/peerstats.20221130: Permission denied
2022-11-30T01:19:50.262 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.262 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.263 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.263 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.264 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/rawstats.20221130: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied
2022-11-30T01:19:50.265 INFO:teuthology.orchestra.run.smithi175.stderr:30 Nov 01:19:50 ntpd[17296]: can't open /var/log/ntpstats/peerstats.20221130: Permission denied
And eventually..
2022-11-30T01:19:56.318 INFO:teuthology.run_tasks:Running task pexec...
2022-11-30T01:19:56.328 INFO:teuthology.task.pexec:Executing custom commands...
2022-11-30T01:19:56.329 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi143.front.sepia.ceph.com
2022-11-30T01:19:56.329 DEBUG:teuthology.orchestra.run.smithi143:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-11-30T01:19:56.331 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi175.front.sepia.ceph.com
2022-11-30T01:19:56.331 DEBUG:teuthology.orchestra.run.smithi175:> TESTDIR=/home/ubuntu/cephtest bash -s
2022-11-30T01:19:56.349 INFO:teuthology.orchestra.run.smithi143.stderr:cp: cannot stat '/etc/containers/registries.conf': No such file or directory
As we've established, this bug only seems to occur on Pacific. The difference between tasks/dashboard tests on main/quincy vs. pacific is that we don't symlink to "supported-random-distros$" in main/quincy. There must be a commit somewhere where we dropped support for ubuntu that just needs to be backported.
tasks/dashboard for main: https://github.com/ceph/ceph/tree/main/qa/suites/rados/dashboard
tasks/dashboard for quincy: https://github.com/ceph/ceph/tree/quincy/qa/suites/rados/dashboard
tasks/dashboard for pacific: https://github.com/ceph/ceph/tree/pacific/qa/suites/rados/dashboard
Updated by Laura Flores over 3 years ago
These two commits pinned dashboard e2e test on centos only:
https://github.com/ceph/ceph/pull/42084
https://github.com/ceph/ceph/pull/42158
Both are from Kefu, so I may ask his opinion on this to make sure we are backporting the right things.
Updated by Laura Flores over 3 years ago
- Status changed from New to Fix Under Review
- Pull request ID set to 49337
I think all that's needed is to remove the random distros facet, so I'm proposing backporting that commit. But I will ask for Kefu's opinion.
Updated by Laura Flores about 3 years ago
- Status changed from Fix Under Review to Resolved
Updated by Upkeep Bot 8 months ago
- Merge Commit set to 708b6e54a7cf826be5858edb64a329871c036e7c
- Fixed In set to v16.2.10-1595-g708b6e54a7c
- Released In set to v16.2.11~16
- Upkeep Timestamp set to 2025-07-14T06:13:39+00:00
Actions