Actions
Bug #70714
openrados/cephadm: CEPHADM_DAEMON_PLACE_FAIL - Failed to extract uid/gid for path /var/lib/grafana
Status:
New
Priority:
Normal
Assignee:
-
Category:
-
Target version:
-
% Done:
0%
Source:
Backport:
tentacle
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Description
/a/yuriw-2025-03-27_15:03:25-rados-wip-yuri7-testing-2025-03-26-1605-distro-default-smithi/8213271
teuthology.log
2025-03-27T17:05:12.815 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: Failed while placing grafana.smithi107 on smithi107: cephadm exited with an error code: 1, stderr: ['Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\nDeploy daemon grafana.smithi107 ...\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana\nstat: stderr Trying to pull quay.io/ceph/grafana:10.4.16...\nstat: stderr Getting image source signatures\nstat: stderr Copying blob sha256:4654044d3b728056007f00717976d87e02598461813e899cba5eb8cb76b25c5b\nstat: stderr Copying blob sha256:54c626fe0d529b8963ba56fa3cc5f9fdc5238986d1b63a2a4ffb964619e030bb\nstat: stderr Copying blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f\nstat: stderr Copying blob sha256:90f5a11ff789bc78fb2bfbf1ba3e3a4a6dc3cbfb70a59b5b1aa399317b58a2df\nstat: stderr Copying blob sha256:66a3d608f3fa52124f8463e9467f170c784abd549e8216aa45c6960b00b4b79b\nstat: stderr Copying blob sha256:403f0da457262de71e12217b22f3cd58e7dac2a813a9dd2a23fa317b8bc3be78\nstat: stderr Copying blob sha256:93c1de5a0286054c877071c4e8a1b800be8409a2e6c7876b3c1b4e63af1f4809\nstat: stderr Copying blob sha256:a47429b3060780d8eef4f2d673f2d1d724c121a524454a6744a54e660f2a522e\nstat: stderr Copying blob sha256:2e57766f290bbc04621bbeb15de7e96aea9ef9bd2680d9e4f2be91e414bc32c5\nstat: stderr Copying blob sha256:37c06f5cf9a9827456c85aa850b4ce3f2ebafa51f975d1a6f75384f54b4e5e5b\nstat: stderr Error: copying system image from manifest list: reading blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f: fetching blob: received unexpected HTTP status: 502 Bad Gateway\nERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana']
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: from='mgr.14162 172.21.15.107:0/2450673418' entity='mgr.smithi107.fvdypl'
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: from='mgr.14162 172.21.15.107:0/2450673418' entity='mgr.smithi107.fvdypl'
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: from='mgr.14162 172.21.15.107:0/2450673418' entity='mgr.smithi107.fvdypl'
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: from='mgr.14162 172.21.15.107:0/2450673418' entity='mgr.smithi107.fvdypl'
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: from='mgr.14162 172.21.15.107:0/2450673418' entity='mgr.smithi107.fvdypl'
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: Deploying daemon prometheus.smithi107 on smithi107
2025-03-27T17:05:12.816 INFO:journalctl@ceph.mon.smithi107.smithi107.stdout:Mar 27 17:05:12 smithi107 ceph-mon[31769]: Health check failed: Failed to place 1 daemon(s) (CEPHADM_DAEMON_PLACE_FAIL)
ceph-mon.smithi107.log.gz
2025-03-27T17:05:11.338+0000 7f93436ea640 20 mon.smithi107@0(leader).mgrstat health checks:
{
"CEPHADM_DAEMON_PLACE_FAIL": {
"severity": "HEALTH_WARN",
"summary": {
"message": "Failed to place 1 daemon(s)",
"count": 1
},
"detail": [
{
"message": "Failed while placing grafana.smithi107 on smithi107: cephadm exited with an error code: 1, stderr: ['Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\\nDeploy daemon grafana.smithi107 ...\\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana\\nstat: stderr Trying to pull quay.io/ceph/grafana:10.4.16...\\nstat: stderr Getting image source signatures\\nstat: stderr Copying blob sha256:4654044d3b728056007f00717976d87e02598461813e899cba5eb8cb76b25c5b\\nstat: stderr Copying blob sha256:54c626fe0d529b8963ba56fa3cc5f9fdc5238986d1b63a2a4ffb964619e030bb\\nstat: stderr Copying blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f\\nstat: stderr Copying blob sha256:90f5a11ff789bc78fb2bfbf1ba3e3a4a6dc3cbfb70a59b5b1aa399317b58a2df\\nstat: stderr Copying blob sha256:66a3d608f3fa52124f8463e9467f170c784abd549e8216aa45c6960b00b4b79b\\nstat: stderr Copying blob sha256:403f0da457262de71e12217b22f3cd58e7dac2a813a9dd2a23fa317b8bc3be78\\nstat: stderr Copying blob sha256:93c1de5a0286054c877071c4e8a1b800be8409a2e6c7876b3c1b4e63af1f4809\\nstat: stderr Copying blob sha256:a47429b3060780d8eef4f2d673f2d1d724c121a524454a6744a54e660f2a522e\\nstat: stderr Copying blob sha256:2e57766f290bbc04621bbeb15de7e96aea9ef9bd2680d9e4f2be91e414bc32c5\\nstat: stderr Copying blob sha256:37c06f5cf9a9827456c85aa850b4ce3f2ebafa51f975d1a6f75384f54b4e5e5b\\nstat: stderr Error: copying system image from manifest list: reading blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f: fetching blob: received unexpected HTTP status: 502 Bad Gateway\\nERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana']"
}
]
}
}
ceph.cephadm.log.gz
2025-03-27T17:05:06.055616+0000 mgr.smithi107.fvdypl (mgr.14162) 35 : cephadm [INF] Deploying daemon grafana.smithi107 on smithi107
2025-03-27T17:05:11.339345+0000 mgr.smithi107.fvdypl (mgr.14162) 38 : cephadm [ERR] Failed while placing grafana.smithi107 on smithi107: cephadm exited with an error code: 1, stderr: ['Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana-smithi107\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\n/usr/bin/podman: stderr Error: no such container ceph-3bcd5ca2-0b2d-11f0-bb99-bd4984dce30f-grafana.smithi107\nDeploy daemon grafana.smithi107 ...\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana\nstat: stderr Trying to pull quay.io/ceph/grafana:10.4.16...\nstat: stderr Getting image source signatures\nstat: stderr Copying blob sha256:4654044d3b728056007f00717976d87e02598461813e899cba5eb8cb76b25c5b\nstat: stderr Copying blob sha256:54c626fe0d529b8963ba56fa3cc5f9fdc5238986d1b63a2a4ffb964619e030bb\nstat: stderr Copying blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f\nstat: stderr Copying blob sha256:90f5a11ff789bc78fb2bfbf1ba3e3a4a6dc3cbfb70a59b5b1aa399317b58a2df\nstat: stderr Copying blob sha256:66a3d608f3fa52124f8463e9467f170c784abd549e8216aa45c6960b00b4b79b\nstat: stderr Copying blob sha256:403f0da457262de71e12217b22f3cd58e7dac2a813a9dd2a23fa317b8bc3be78\nstat: stderr Copying blob sha256:93c1de5a0286054c877071c4e8a1b800be8409a2e6c7876b3c1b4e63af1f4809\nstat: stderr Copying blob sha256:a47429b3060780d8eef4f2d673f2d1d724c121a524454a6744a54e660f2a522e\nstat: stderr Copying blob sha256:2e57766f290bbc04621bbeb15de7e96aea9ef9bd2680d9e4f2be91e414bc32c5\nstat: stderr Copying blob sha256:37c06f5cf9a9827456c85aa850b4ce3f2ebafa51f975d1a6f75384f54b4e5e5b\nstat: stderr Error: copying system image from manifest list: reading blob sha256:5616aa02b86cd5e8daa20f6c32e6f52e750e64a257258a319dfba14a40f42b5f: fetching blob: received unexpected HTTP status: 502 Bad Gateway\nERROR: Failed to extract uid/gid for path /var/lib/grafana: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/ceph/grafana:10.4.16 -e NODE_NAME=smithi107 quay.io/ceph/grafana:10.4.16 -c %u %g /var/lib/grafana']
Updated by Laura Flores 11 months ago
/a/lflores-2025-04-11_19:10:45-rados-wip-lflores-testing-3-2025-04-11-1140-distro-default-smithi/8236042
Updated by Nitzan Mordechai 10 months ago
we starting to see that error in few other runs: /a/teuthology-2025-05-18_21:00:02-rados-squid-distro-default-smithi
2025-05-19T03:38:09.766+0000 7f154812e640 0 [cephadm DEBUG cephadm.serve] err: Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-19f5d9ac-3462-11f0-86fc-adfe0268badd-prometheus-smithi0
55
/usr/bin/docker: stdout
/usr/bin/docker: stderr Error response from daemon: No such container: ceph-19f5d9ac-3462-11f0-86fc-adfe0268badd-prometheus-smithi055
Non-zero exit code 1 from /usr/bin/docker container inspect --format {{.State.Status}} ceph-19f5d9ac-3462-11f0-86fc-adfe0268badd-prometheus.smithi055
/usr/bin/docker: stdout
/usr/bin/docker: stderr Error response from daemon: No such container: ceph-19f5d9ac-3462-11f0-86fc-adfe0268badd-prometheus.smithi055
Deploy daemon prometheus.smithi055 ...
Non-zero exit code 125 from /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.51.0 -e NODE_NAME=smithi055 q
uay.io/prometheus/prometheus:v2.51.0 -c %u %g /etc/prometheus
stat: stderr Unable to find image 'quay.io/prometheus/prometheus:v2.51.0' locally
stat: stderr docker: Error response from daemon: Get "https://quay.io/v2/prometheus/prometheus/manifests/sha256:5ccad477d0057e62a7cd1981ffcc43785ac10c5a35522dc207466ff7e7ec845f": net/http: TLS handshake timeout.
stat: stderr See 'docker run --help'.
ERROR: Failed to extract uid/gid for path /etc/prometheus: Failed command: /usr/bin/docker run --rm --ipc=host --stop-signal=SIGTERM --ulimit nofile=1048576 --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/prometheus:v2.51.0 -e NODE_NAME=smithi055 quay.io/prometheus/prometheus:v2.51.0 -c %u %g /etc/prometheus
Updated by Laura Flores 10 months ago
/a/yuriw-2025-04-29_17:45:14-rados-wip-yuri4-testing-2025-04-29-0801-squid-distro-default-smithi/8265208
2025-04-29T21:03:01.823+0000 7f6bd03a2640 20 mon.a@0(leader).mgrstat health checks:
{
"CEPHADM_DAEMON_PLACE_FAIL": {
"severity": "HEALTH_WARN",
"summary": {
"message": "Failed to place 1 daemon(s)",
"count": 1
},
"detail": [
{
"message": "Failed while placing alertmanager.a on smithi087: cephadm exited with an error code: 1, stderr: Non-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-520d611e-253c-11f0-bba5-bd4984dce30f-alertmanager-a\n/usr/bin/podman: stderr Error: no such container ceph-520d611e-253c-11f0-bba5-bd4984dce30f-alertmanager-a\nNon-zero exit code 125 from /usr/bin/podman container inspect --format {{.State.Status}} ceph-520d611e-253c-11f0-bba5-bd4984dce30f-alertmanager.a\n/usr/bin/podman: stderr Error: no such container ceph-520d611e-253c-11f0-bba5-bd4984dce30f-alertmanager.a\nDeploy daemon alertmanager.a ...\nNon-zero exit code 125 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi087 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/alertmanager\nstat: stderr Trying to pull quay.io/prometheus/alertmanager:v0.25.0...\nstat: stderr Getting image source signatures\nstat: stderr Copying blob sha256:d71d159599c38915c22c878fdb13c857684102338f6f33d3f0011e36ad117c04\nstat: stderr Copying blob sha256:b08a0a8262352677ce3e10b697ebda40ffffcfb2cc4dd66a93fc220b940801f5\nstat: stderr Copying blob sha256:c4dc43cc86853f40d99d6199570e9f823856fa3e7992dcd7ebe94fa4d32a0ae6\nstat: stderr Copying blob sha256:05d21abf0535766aaa32ec4541e1213912944d7dea17e40e71a84177f9000b68\nstat: stderr Copying blob sha256:aff850a11e318220e60d82e705674a12e50fb96de4644ab665b2865d7c783796\nstat: stderr Copying blob sha256:6c477a8cc220cbe4c1ffdd9cb505ca82284292c367a03335c5bd590ff0c651fc\nstat: stderr Error: copying system image from manifest list: reading blob sha256:b08a0a8262352677ce3e10b697ebda40ffffcfb2cc4dd66a93fc220b940801f5: fetching blob: received unexpected HTTP status: 502 Bad Gateway\nNon-zero exit code 1 from /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi087 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/prometheus\nstat: stderr Trying to pull quay.io/prometheus/alertmanager:v0.25.0...\nstat: stderr Getting image source signatures\nstat: stderr Copying blob sha256:b08a0a8262352677ce3e10b697ebda40ffffcfb2cc4dd66a93fc220b940801f5\nstat: stderr Copying blob sha256:6c477a8cc220cbe4c1ffdd9cb505ca82284292c367a03335c5bd590ff0c651fc\nstat: stderr Copying blob sha256:05d21abf0535766aaa32ec4541e1213912944d7dea17e40e71a84177f9000b68\nstat: stderr Copying blob sha256:d71d159599c38915c22c878fdb13c857684102338f6f33d3f0011e36ad117c04\nstat: stderr Copying blob sha256:c4dc43cc86853f40d99d6199570e9f823856fa3e7992dcd7ebe94fa4d32a0ae6\nstat: stderr Copying blob sha256:aff850a11e318220e60d82e705674a12e50fb96de4644ab665b2865d7c783796\nstat: stderr Copying config sha256:c8568f914cd25b2062c44e9f79f9c18da6e3b85fe0c47a12a2191c61426c2b19\nstat: stderr Writing manifest to image destination\nstat: stderr stat: can't stat '/etc/prometheus': No such file or directory\nERROR: Failed to extract uid/gid for path /etc/prometheus: Failed command: /usr/bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=quay.io/prometheus/alertmanager:v0.25.0 -e NODE_NAME=smithi087 quay.io/prometheus/alertmanager:v0.25.0 -c %u %g /etc/prometheus"
}
]
}
}
Updated by Laura Flores 10 months ago
- Backport set to tentacle
/a/teuthology-2025-05-18_22:00:03-rados-tentacle-distro-default-smithi/8288417
Actions