Actions
Bug #74577
opentest_iscsi_setup.sh - No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1
% Done:
0%
Source:
Backport:
Regression:
No
Severity:
3 - minor
Reviewed:
Affected Versions:
ceph-qa-suite:
Pull request ID:
Tags (freeform):
Merge Commit:
Fixed In:
Released In:
Upkeep Timestamp:
Tags:
Description
/a/nmordech-2026-01-26_18:32:32-rados:cephadm-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/18790/teuthology.log
2026-01-26T19:15:04.368 INFO:tasks.workunit.client.0.trial044.stderr:+ container_gwcli /disks create pool=foo image=disk_2 size=2G
2026-01-26T19:15:04.368 INFO:tasks.workunit.client.0.trial044.stderr:+ sudo podman exec -it a60aa7b1397f gwcli /disks create pool=foo image=disk_2 size=2G
2026-01-26T19:15:04.925 INFO:tasks.workunit.client.0.trial044.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts create iqn.1994-05.com.redhat:client1
2026-01-26T19:15:04.925 INFO:tasks.workunit.client.0.trial044.stderr:+ sudo podman exec -it a60aa7b1397f gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts create iqn.1994-05.com.redhat:client1
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.14283 -' entity='client.iscsi.foo.trial044.tjeomj' cmd=[{"prefix": "service status", "format": "json"}]: dispatch
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/713268970' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "version"} : dispatch
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/62817200' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "osd dump", "format": "json"} : dispatch
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/1205791776' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "status", "format": "json"} : dispatch
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/2295588056' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "status", "format": "json"} : dispatch
2026-01-26T19:15:05.070 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/1998623723' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "df", "format": "json"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/3470394720' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "df", "format": "json"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/3712460921' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "version"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/3984617087' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "osd dump", "format": "json"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/1888630168' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "status", "format": "json"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/3183434968' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "status", "format": "json"} : dispatch
2026-01-26T19:15:05.071 INFO:journalctl@ceph.mon.a.trial044.stdout:Jan 26 19:15:04 trial044 ceph-mon[18960]: from='client.? 10.20.193.44:0/1490697474' entity='client.iscsi.foo.trial044.tjeomj' cmd={"prefix": "df", "format": "json"} : dispatch
2026-01-26T19:15:05.387 INFO:tasks.workunit.client.0.trial044.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1 auth username=adminadmin password=adminadminadmin
2026-01-26T19:15:05.387 INFO:tasks.workunit.client.0.trial044.stderr:+ sudo podman exec -it a60aa7b1397f gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1 auth username=adminadmin password=adminadminadmin
2026-01-26T19:15:05.804 INFO:tasks.workunit.client.0.trial044.stdout:No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1
2026-01-26T19:15:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 255
2026-01-26T19:15:05.840 INFO:tasks.workunit:Stopping ['cephadm/test_iscsi_pids_limit.sh', 'cephadm/test_iscsi_etc_hosts.sh', 'cephadm/test_iscsi_setup.sh'] on client.0...
Updated by Yaarit Hatuka about 2 months ago
- Related to QA Run #74540: wip-rocky10-branch-of-the-day-2026-01-23-1769128778 added
Updated by Nitzan Mordechai about 2 months ago
/a/nmordech-2026-01-26_18:32:32-rados:cephadm-wip-rocky10-branch-of-the-day-2026-01-23-1769128778-distro-default-trial/18790/teuthology.log shows right after the create that the path is not exist:
2026-01-26T19:15:05.387 INFO:tasks.workunit.client.0.trial044.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1 auth username=adminadmin password=adminadminadmin 2026-01-26T19:15:05.387 INFO:tasks.workunit.client.0.trial044.stderr:+ sudo podman exec -it a60aa7b1397f gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1 auth username=adminadmin password=adminadminadmin 2026-01-26T19:15:05.804 INFO:tasks.workunit.client.0.trial044.stdout:No such path /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts/iqn.1994-05.com.redhat:client1 2026-01-26T19:15:05.839 DEBUG:teuthology.orchestra.run:got remote process result: 255
i checked journalctl.log.gz and there are few http errors:
Jan 26 19:15:05 trial044 ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460-iscsi-foo-trial044-tjeomj[50302]: debug ::1 - - [26/Jan/2026 19:15:05] "PUT /api/client/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/iqn.1994-05.com.redhat:client1 HTTP/1.1" 400 -
and cephadm log show:
2026-01-26 19:14:22,584 7fbbfd0a7e00 INFO Creating ceph-iscsi config... 2026-01-26 19:14:22,584 7fbbfd0a7e00 INFO Write file: /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/iscsi.foo.trial044.tjeomj/iscsi-gateway.cfg 2026-01-26 19:14:22,584 7fbbfd0a7e00 INFO Write file: /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/iscsi.foo.trial044.tjeomj/tcmu-runner-entrypoint.sh 2026-01-26 19:14:22,734 7fbbfd0a7e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460@iscsi.foo.trial044.tjeomj 2026-01-26 19:14:22,734 7fbbfd0a7e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460@iscsi.foo.trial044.tjeomj.service: Unit ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460@iscsi.foo.trial044.tjeomj.service not loaded. 2026-01-26 19:14:22,753 7fbbfd0a7e00 DEBUG Non-zero exit code 1 from systemctl reset-failed ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460-sidecar@iscsi.foo.trial044.tjeomj:tcmu.service 2026-01-26 19:14:22,753 7fbbfd0a7e00 DEBUG systemctl: stderr Failed to reset failed state of unit ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460-sidecar@iscsi.foo.trial044.tjeomj:tcmu.service: Unit ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460-sidecar@iscsi.foo.trial044.tjeomj:tcmu.service not loaded. 2026-01-26 19:14:22,871 7fbbfd0a7e00 DEBUG systemctl: stderr Created symlink /etc/systemd/system/ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460.target.wants/ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460@iscsi.foo.trial044.tjeomj.service → /etc/systemd/system/ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460@.service. 2026-01-26 19:14:23,022 7fbbfd0a7e00 DEBUG firewalld.service is not enabled 2026-01-26 19:14:23,022 7fbbfd0a7e00 DEBUG Not possible to open ports <[5000, 3260]>. firewalld.service is not available 2026-01-26 19:14:23,325 7f9fa243fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3', '--timeout', '895', 'ls'] 2026-01-26 19:14:59,810 7fa2bb202e00 DEBUG -------------------------------------------------------------------------------- cephadm ['ls'] 2026-01-26 19:15:00,384 7fce78be7e00 DEBUG -------------------------------------------------------------------------------- cephadm ['enter', '--name', 'iscsi.foo.trial044.tjeomj', '--', 'cat', '/etc/hosts'] 2026-01-26 19:15:00,402 7fce78be7e00 DEBUG Using default config /etc/ceph/ceph.conf 2026-01-26 19:15:00,402 7fce78be7e00 INFO Inferring fsid 0921deeb-faeb-11f0-a3a7-d404e6e7d460 2026-01-26 19:15:00,424 7fce78be7e00 DEBUG /bin/podman: stdout running 2026-01-26 19:15:00,424 7fce78be7e00 DEBUG Running command (timeout=None): /bin/podman exec -i ceph-0921deeb-faeb-11f0-a3a7-d404e6e7d460-iscsi-foo-trial044-tjeomj cat /etc/hosts 2026-01-26 19:15:01,115 7fd976850e00 DEBUG -------------------------------------------------------------------------------- cephadm ['ls', '--no-detail'] 2026-01-26 19:15:01,233 7f1f00fd8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['shell', '--', 'ceph', 'orch', 'ps', '--daemon-id', 'foo.trial044.tjeomj', '-f', 'json'] 2026-01-26 19:15:01,252 7f1f00fd8e00 DEBUG Using default config /etc/ceph/ceph.conf 2026-01-26 19:15:01,252 7f1f00fd8e00 INFO Inferring fsid 0921deeb-faeb-11f0-a3a7-d404e6e7d460 2026-01-26 19:15:01,253 7f1f00fd8e00 INFO Inferring config /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config 2026-01-26 19:15:01,439 7f1f00fd8e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 -e NODE_NAME=trial044 -v /var/run/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/run/ceph:z -v /var/log/c eph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/config/ceph.client.admin. keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 orch ps --daemon-id foo.trial044.tjeomj -f json 2026-01-26 19:15:01,800 7f9757e8fe00 DEBUG -------------------------------------------------------------------------------- cephadm ['shell', '--', 'ceph', 'orch', 'host', 'ls', '--format', 'json'] 2026-01-26 19:15:01,819 7f9757e8fe00 DEBUG Using default config /etc/ceph/ceph.conf 2026-01-26 19:15:01,820 7f9757e8fe00 INFO Inferring fsid 0921deeb-faeb-11f0-a3a7-d404e6e7d460 2026-01-26 19:15:01,820 7f9757e8fe00 INFO Inferring config /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config 2026-01-26 19:15:02,011 7f9757e8fe00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 -e NODE_NAME=trial044 -v /var/run/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/run/ceph:z -v /var/log/c eph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/config/ceph.client.admin. keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 orch host ls --format json 2026-01-26 19:15:02,395 7f4621de8e00 DEBUG -------------------------------------------------------------------------------- cephadm ['shell', '--', 'ceph', 'orch', 'ls', 'iscsi', '--format', 'json'] 2026-01-26 19:15:02,414 7f4621de8e00 DEBUG Using default config /etc/ceph/ceph.conf 2026-01-26 19:15:02,415 7f4621de8e00 INFO Inferring fsid 0921deeb-faeb-11f0-a3a7-d404e6e7d460 2026-01-26 19:15:02,415 7f4621de8e00 INFO Inferring config /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config 2026-01-26 19:15:02,611 7f4621de8e00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 -e NODE_NAME=trial044 -v /var/run/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config:/etc/ceph/ceph.conf:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/config/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph@sha256:91cae0a98ff138bf404f0c27e753f19d6d7ad1d9e0e36d77100ce5dd29a11eb3 orch ls iscsi --format json 2026-01-26 19:15:06,345 7f4e4ac6de00 DEBUG -------------------------------------------------------------------------------- cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:b62a951ffc48a50e41d23e63ed6b312afb1c1621-rockylinux-10', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '0921deeb-faeb-11f0-a3a7-d404e6e7d460', '--', 'ceph', 'mgr', 'module', 'disable', 'cephadm'] 2026-01-26 19:15:06,365 7f4e4ac6de00 INFO Inferring config /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config 2026-01-26 19:15:06,365 7f4e4ac6de00 DEBUG Using specified fsid: 0921deeb-faeb-11f0-a3a7-d404e6e7d460 2026-01-26 19:15:06,365 7f4e4ac6de00 DEBUG Using specified config: /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config 2026-01-26 19:15:06,365 7f4e4ac6de00 DEBUG Running command (timeout=None): /bin/podman run --rm --ipc=host --net=host --privileged --group-add=disk --init -i -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:b62a951ffc48a50e41d23e63ed6b312afb1c1621-rockylinux-10 -e NODE_NAME=trial044 -v /var/run/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/run/ceph:z -v /var/log/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460:/var/log/ceph:z -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/crash:/var/lib/ceph/crash:z -v /run/systemd/journal:/run/systemd/journal -v /etc/hosts:/etc/hosts:ro -v /var/lib/ceph/0921deeb-faeb-11f0-a3a7-d404e6e7d460/mon.a/config:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:b62a951ffc48a50e41d23e63ed6b312afb1c1621-rockylinux-10 mgr module disable cephadm 2026-01-26 19:15:28,009 7fb4b52b6e00 DEBUG --------------------------------------------------------------------------------
can someone from cephadm can take a look?
maybe its infra issue?
2026-01-26 19:14:23,022 7fbbfd0a7e00 DEBUG firewalld.service is not enabled 2026-01-26 19:14:23,022 7fbbfd0a7e00 DEBUG Not possible to open ports <[5000, 3260]>. firewalld.service is not available
Updated by Yaarit Hatuka about 2 months ago
- Blocks Bug #73930: ceph-mgr modules rely on deprecated python subinterpreters added
Updated by David Galloway about 2 months ago
firewalld is installed on Rocky 10. If the test needs it, the test should enable it.
Updated by Laura Flores about 1 month ago
/a/yaarit-2026-02-05_17:05:15-rados:cephadm-wip-rocky10-branch-of-the-day-2026-02-03-1770151121-distro-default-trial/36997
Updated by Laura Flores about 1 month ago · Edited
I'm not totally convinced that this is a Rocky10 issue.
On all the main runs I can see, this type of iscsi test fails from a different known bug. For instance, here is an iscsi test from one of our latest main batches:
https://pulpito.ceph.com/skanta-2026-02-04_08:34:03-rados-wip-bharath2-testing-2026-02-03-0542-distro-default-trial/34139
This test failed from a known issue: https://tracker.ceph.com/issues/68586
This bug only occurs with "agent/on". But the test that Nitzan ran had "agent/off".
I can't find a test in our history that ran on main with "agent/off" to compare, so I ran some here so we can check. Then we'll know for sure if this is specific to Rocky10 or not:
https://pulpito.ceph.com/lflores-2026-02-06_00:17:52-orch:cephadm-main-distro-default-trial/
Updated by Laura Flores about 1 month ago
- Related to QA Run #74752: wip-rocky10-branch-of-the-day-2026-02-03-1770151121 added
Updated by Redouane Kachach Elhicou about 1 month ago
Laura, it the jobs you scheduled ran successfully, there's only one dead job but it seems caused by a scheduling issue (node has not enough disks):
failure_reason: '["Wanted 2 disks of ~1700 GB (rotational=False), but only matched
1: [''nvme1n1'']"]'
Updated by Redouane Kachach Elhicou about 1 month ago
I created a PR with some small adjustments of the iscsi script with the goal to improve the logging in case of failures and added some extra intermediate checks. It would be great if you can re-run the job with my modifications and see if the error persists (or not). In the worst case we should get better debugging. Thanks.
Updated by Laura Flores about 1 month ago · Edited
Redouane Kachach Elhichou wrote in #note-9:
Laura, it the jobs you scheduled ran successfully, there's only one dead job but it seems caused by a scheduling issue (node has not enough disks):
failure_reason: '["Wanted 2 disks of ~1700 GB (rotational=False), but only matched
1: [''nvme1n1'']"]'
Yeah, at this time, this means that the test is indeed passing on main and failing with rocky10.
Redouane Kachach Elhichou wrote in #note-10:
I created a PR with some small adjustments of the iscsi script with the goal to improve the logging in case of failures and added some extra intermediate checks. It would be great if you can re-run the job with my modifications and see if the error persists (or not). In the worst case we should get better debugging. Thanks.
Yes, I reran the exact job that failed on the latest rocky10 branch of the day here with your commit added on top of the suite branch:
https://pulpito.ceph.com/lflores-2026-02-06_21:15:53-rados:cephadm-wip-rocky10-branch-of-the-day-2026-02-03-1770151121-distro-default-trial/
./teuthology/virtualenv/bin/teuthology-suite --priority 11 -m trial --suite-repo https://github.com/ljflores/ceph.git --ceph-repo https://git.ceph.com/ceph-ci.git -c wip-rocky10-branch-of-the-day-2026-02-03-1770151121 -r yaarit-2026-02-05_17:05:15-rados:cephadm-wip-rocky10-branch-of-the-day-2026-02-03-1770151121-distro-default-trial --filter-all "cephadm/workunits/{0-distro/rocky_10 agent/off mon_election/connectivity task/test_iscsi_container/{centos_9.stream test_iscsi_container}}" --force-priority --newest 10 --suite-branch rocky101 $(pwd)/cephadm-rocky10.yaml -N 3
All three tests failed from the same bug. The teuthology log should contain some more useful debug information.
From this experiment, I think we can conclude that this issue is specific to Rocky10.
Updated by Laura Flores about 1 month ago
- Assignee set to Redouane Kachach Elhicou
Updated by Nitzan Mordechai about 1 month ago
/a/yaarit-2026-02-08_02:25:21-rados-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/39911
Updated by Nitzan Mordechai about 1 month ago
/a/nmordech-2026-02-09_08:34:53-rados-wip-rocky10-branch-of-the-day-2026-02-06-1770413686-distro-default-trial/41227
Updated by Nitzan Mordechai about 1 month ago
/a/yaarit-2026-02-10_02:32:02-rados:cephadm-wip-rocky10-branch-of-the-day-2026-02-09-1770676549-distro-default-trial/42012
Updated by David Galloway about 1 month ago
Here's your extra debugging info
2026-02-10T13:34:42.380 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways ls
2026-02-10T13:34:42.380 INFO:tasks.workunit.client.0.trial128.stderr:+ grep -qF trial128
2026-02-10T13:34:42.720 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:42 trial128 ceph-mon[21317]: pgmap v91: 33 pgs: 33 active+clean; 578 KiB data, 82 MiB used, 2.1 TiB / 2.1 TiB avail; 2.6 KiB/s rd, 4 op/s
2026-02-10T13:34:42.806 INFO:tasks.workunit.client.0.trial128.stderr:+ sleep 1
2026-02-10T13:34:43.807 INFO:tasks.workunit.client.0.trial128.stderr:+ (( i++ ))
2026-02-10T13:34:43.807 INFO:tasks.workunit.client.0.trial128.stderr:+ (( i<=retries ))
2026-02-10T13:34:43.807 INFO:tasks.workunit.client.0.trial128.stderr:+ return 1
2026-02-10T13:34:43.807 INFO:tasks.workunit.client.0.trial128.stderr:+ echo 'Gateway trial128 not visible in gwcli after create; failing early.'
2026-02-10T13:34:43.807 INFO:tasks.workunit.client.0.trial128.stdout:Gateway trial128 not visible in gwcli after create; failing early.
2026-02-10T13:34:43.808 INFO:tasks.workunit.client.0.trial128.stderr:+ dump_gwcli_state
2026-02-10T13:34:43.808 INFO:tasks.workunit.client.0.trial128.stderr:+ set +e
2026-02-10T13:34:43.808 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli / ls
2026-02-10T13:34:43.808 INFO:tasks.workunit.client.0.trial128.stderr:+ sudo podman exec -i c900ea087237 gwcli / ls
2026-02-10T13:34:43.813 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:43 trial128 ceph-mon[21317]: from='client.? 10.20.193.128:0/3432166239' entity='client.iscsi.foo.trial128.pnarku' cmd={"prefix": "version"} : dispatch
2026-02-10T13:34:43.813 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:43 trial128 ceph-mon[21317]: from='client.? 10.20.193.128:0/2067524167' entity='client.iscsi.foo.trial128.pnarku' cmd={"prefix": "osd dump", "format": "json"} : dispatch
2026-02-10T13:34:43.814 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:43 trial128 ceph-mon[21317]: from='client.? 10.20.193.128:0/546650941' entity='client.iscsi.foo.trial128.pnarku' cmd={"prefix": "status", "format": "json"} : dispatch
2026-02-10T13:34:43.814 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:43 trial128 ceph-mon[21317]: from='client.? 10.20.193.128:0/4135707493' entity='client.iscsi.foo.trial128.pnarku' cmd={"prefix": "status", "format": "json"} : dispatch
2026-02-10T13:34:43.814 INFO:journalctl@ceph.mon.a.trial128.stdout:Feb 10 13:34:43 trial128 ceph-mon[21317]: from='client.? 10.20.193.128:0/1465506552' entity='client.iscsi.foo.trial128.pnarku' cmd={"prefix": "df", "format": "json"} : dispatch
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout:o- / ......................................................................................................................... [...]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: o- cluster ......................................................................................................... [Clusters: 1]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: | o- ceph ............................................................................................................ [HEALTH_OK]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: | o- pools .......................................................................................................... [Pools: 2]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: | | o- .mgr ................................................................. [(x3), Commit: 0.00Y/697275200K (0%), Used: 1740K]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: | | o- foo .................................................................... [(x3), Commit: 0.00Y/697275200K (0%), Used: 24K]
2026-02-10T13:34:44.201 INFO:tasks.workunit.client.0.trial128.stdout: | o- topology ................................................................................................ [OSDs: 3,MONs: 1]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- disks ....................................................................................................... [0.00Y, Disks: 0]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- iscsi-targets ............................................................................... [DiscoveryAuth: None, Targets: 1]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ......................................................... [Auth: None, Gateways: 0]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- disks .......................................................................................................... [Disks: 0]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- gateways ............................................................................................ [Up: 0/0, Portals: 0]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- host-groups .................................................................................................. [Groups : 0]
2026-02-10T13:34:44.202 INFO:tasks.workunit.client.0.trial128.stdout: o- hosts ....................................................................................... [Auth: ACL_ENABLED, Hosts: 0]
2026-02-10T13:34:44.232 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli /iscsi-targets ls
2026-02-10T13:34:44.232 INFO:tasks.workunit.client.0.trial128.stderr:+ sudo podman exec -i c900ea087237 gwcli /iscsi-targets ls
2026-02-10T13:34:44.622 INFO:tasks.workunit.client.0.trial128.stdout:o- iscsi-targets ................................................................................. [DiscoveryAuth: None, Targets: 1]
2026-02-10T13:34:44.622 INFO:tasks.workunit.client.0.trial128.stdout: o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ........................................................... [Auth: None, Gateways: 0]
2026-02-10T13:34:44.623 INFO:tasks.workunit.client.0.trial128.stdout: o- disks ............................................................................................................ [Disks: 0]
2026-02-10T13:34:44.623 INFO:tasks.workunit.client.0.trial128.stdout: o- gateways .............................................................................................. [Up: 0/0, Portals: 0]
2026-02-10T13:34:44.623 INFO:tasks.workunit.client.0.trial128.stdout: o- host-groups .................................................................................................... [Groups : 0]
2026-02-10T13:34:44.623 INFO:tasks.workunit.client.0.trial128.stdout: o- hosts ......................................................................................... [Auth: ACL_ENABLED, Hosts: 0]
2026-02-10T13:34:44.659 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ls
2026-02-10T13:34:44.659 INFO:tasks.workunit.client.0.trial128.stderr:+ sudo podman exec -i c900ea087237 gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ls
...
2026-02-10T13:34:45.054 INFO:tasks.workunit.client.0.trial128.stdout:o- iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw ............................................................. [Auth: None, Gateways: 0]
2026-02-10T13:34:45.054 INFO:tasks.workunit.client.0.trial128.stdout: o- disks .............................................................................................................. [Disks: 0]
2026-02-10T13:34:45.054 INFO:tasks.workunit.client.0.trial128.stdout: o- gateways ................................................................................................ [Up: 0/0, Portals: 0]
2026-02-10T13:34:45.054 INFO:tasks.workunit.client.0.trial128.stdout: o- host-groups ...................................................................................................... [Groups : 0]
2026-02-10T13:34:45.054 INFO:tasks.workunit.client.0.trial128.stdout: o- hosts ........................................................................................... [Auth: ACL_ENABLED, Hosts: 0]
2026-02-10T13:34:45.129 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways ls
2026-02-10T13:34:45.129 INFO:tasks.workunit.client.0.trial128.stderr:+ sudo podman exec -i c900ea087237 gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/gateways ls
2026-02-10T13:34:45.521 INFO:tasks.workunit.client.0.trial128.stdout:o- gateways .................................................................................................. [Up: 0/0, Portals: 0]
2026-02-10T13:34:45.555 INFO:tasks.workunit.client.0.trial128.stderr:+ container_gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts ls
2026-02-10T13:34:45.555 INFO:tasks.workunit.client.0.trial128.stderr:+ sudo podman exec -i c900ea087237 gwcli /iscsi-targets/iqn.2003-01.com.redhat.iscsi-gw:iscsi-igw/hosts ls
...
2026-02-10T13:34:45.945 INFO:tasks.workunit.client.0.trial128.stdout:o- hosts ............................................................................................. [Auth: ACL_ENABLED, Hosts: 0]
2026-02-10T13:34:45.976 DEBUG:teuthology.orchestra.run:got remote process result: 1
Updated by Nitzan Mordechai about 1 month ago
/a/nmordech-2026-02-17_05:46:21-rados-wip-rocky10-branch-of-the-day-2026-02-13-1771001916-distro-default-trial/
['51861', '51731']
Updated by Nitzan Mordechai 27 days ago
a/nmordech-2026-02-21_21:52:07-rados:cephadm-wip-rocky10-branch-of-the-day-2026-02-21-1771692874-tentacle-distro-default-trial/63013
Actions