Skip to content

qa/cephadm: Add iSCSI#35543

Merged
sebastian-philipp merged 3 commits intoceph:masterfrom
sebastian-philipp:qa-cephadm-iscsi
Sep 11, 2020
Merged

qa/cephadm: Add iSCSI#35543
sebastian-philipp merged 3 commits intoceph:masterfrom
sebastian-philipp:qa-cephadm-iscsi

Conversation

@sebastian-philipp
Copy link
Contributor

Signed-off-by: Sebastian Wagner sebastian.wagner@suse.com

Checklist

  • References tracker ticket
  • Updates documentation if necessary
  • Includes tests for new functionality or reproducer for bug

Show available Jenkins commands
  • jenkins retest this please
  • jenkins test classic perf
  • jenkins test crimson perf
  • jenkins test signed
  • jenkins test make check
  • jenkins test make check arm64
  • jenkins test submodules
  • jenkins test dashboard
  • jenkins test dashboard backend
  • jenkins test docs
  • jenkins render docs
  • jenkins test ceph-volume all
  • jenkins test ceph-volume tox

@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jun 12, 2020

http://pulpito.ceph.com/swagner-2020-06-11_14:18:54-rados:cephadm-wip-swagner-testing-2020-06-11-1428-distro-basic-smithi/

close:

2020-06-11T23:30:06.355 INFO:teuthology.orchestra.run.smithi184.stdout:NAME            RUNNING  REFRESHED  AGE   PLACEMENT                        IMAGE NAME                                                     IMAGE ID
2020-06-11T23:30:06.356 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager        1/1  50s ago    110s  count:1 smithi184=a              docker.io/prom/alertmanager:v0.20.0                            0881eb8f169f
2020-06-11T23:30:06.356 INFO:teuthology.orchestra.run.smithi184.stdout:grafana             1/1  0s ago     101s  count:1 smithi082=a              docker.io/ceph/ceph-grafana:latest                             87a51ecf0b1c
2020-06-11T23:30:06.356 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.a             0/0  0s ago     -     <unmanaged>                      quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  <unknown>
2020-06-11T23:30:06.356 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.iscsi         0/1  -          -     count:1 smithi082=a              <unknown>                                                      <unknown>
2020-06-11T23:30:06.357 INFO:teuthology.orchestra.run.smithi184.stdout:mgr                 1/2  0s ago     5m    count:2 smithi082=x              quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c
2020-06-11T23:30:06.357 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b               3/0  50s ago    -     <unmanaged>                      quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c
2020-06-11T23:30:06.357 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter       2/2  50s ago    114s  count:2 smithi184=a,smithi082=b  docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf
2020-06-11T23:30:06.357 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4               8/0  50s ago    -     <unmanaged>                      quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c
2020-06-11T23:30:06.357 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus          1/1  0s ago     118s  count:1 smithi082=a              docker.io/prom/prometheus:v2.18.1                              de242295e225
2020-06-11T23:30:06.358 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.realm.zone      1/1  50s ago    2m    count:1 smithi184=realm.zone.a   quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c

2020-06-11T23:30:03.462 INFO:teuthology.orchestra.run.smithi184.stdout:NAME              HOST       STATUS         REFRESHED  AGE   VERSION               IMAGE NAME                                                     IMAGE ID      CONTAINER ID
2020-06-11T23:30:03.462 INFO:teuthology.orchestra.run.smithi184.stdout:alertmanager.a    smithi184  running (80s)  47s ago    80s   0.20.0                docker.io/prom/alertmanager:v0.20.0                            0881eb8f169f  1534def0ea30
2020-06-11T23:30:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:grafana.a         smithi082  running (59s)  1s ago     59s   6.6.2                 docker.io/ceph/ceph-grafana:latest                             87a51ecf0b1c  5730f6e615ba
2020-06-11T23:30:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:iscsi.a           smithi082  unknown        1s ago     117s  <unknown>             quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  <unknown>     <unknown>
2020-06-11T23:30:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:mgr.x             smithi082  running (5m)   1s ago     5m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  d80b9de261ff
2020-06-11T23:30:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:mon.a             smithi184  running (7m)   47s ago    7m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  4b72c8dac9ea
2020-06-11T23:30:03.463 INFO:teuthology.orchestra.run.smithi184.stdout:mon.b             smithi082  running (5m)   1s ago     5m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  a59f50887212
2020-06-11T23:30:03.464 INFO:teuthology.orchestra.run.smithi184.stdout:mon.c             smithi184  running (5m)   47s ago    5m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  b5b48398b613
2020-06-11T23:30:03.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.a   smithi184  running (89s)  47s ago    93s   0.18.1                docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf  d12054e6937c
2020-06-11T23:30:03.464 INFO:teuthology.orchestra.run.smithi184.stdout:node-exporter.b   smithi082  running (88s)  1s ago     92s   0.18.1                docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf  4433591bf7e7
2020-06-11T23:30:03.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.0             smithi184  running (4m)   47s ago    4m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  fc189d79cb3e
2020-06-11T23:30:03.464 INFO:teuthology.orchestra.run.smithi184.stdout:osd.1             smithi184  running (4m)   47s ago    4m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  bcb45dee31c0
2020-06-11T23:30:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.2             smithi184  running (3m)   47s ago    3m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  9430006cd20d
2020-06-11T23:30:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.3             smithi184  running (3m)   47s ago    3m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  34bdb147f467
2020-06-11T23:30:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.4             smithi082  running (3m)   1s ago     3m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  034185f326d7
2020-06-11T23:30:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.5             smithi082  running (2m)   1s ago     2m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  09bc301a933e
2020-06-11T23:30:03.465 INFO:teuthology.orchestra.run.smithi184.stdout:osd.6             smithi082  running (2m)   1s ago     2m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  0f7bea4f296e
2020-06-11T23:30:03.466 INFO:teuthology.orchestra.run.smithi184.stdout:osd.7             smithi082  running (2m)   1s ago     2m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  b2dee1e7ea62
2020-06-11T23:30:03.466 INFO:teuthology.orchestra.run.smithi184.stdout:prometheus.a      smithi082  running (97s)  1s ago     97s   2.18.1                docker.io/prom/prometheus:v2.18.1                              de242295e225  583c5e9a3fd9
2020-06-11T23:30:03.466 INFO:teuthology.orchestra.run.smithi184.stdout:rgw.realm.zone.a  smithi184  running (2m)   47s ago    2m    16.0.0-2453-g2f281c3  quay.io/ceph-ci/ceph:2f281c3338231097e63f3d2324b211e9c428ffe9  0fc030a24b3c  e2a03f7c9cfa

Comment on lines +482 to +546
(
# fixed daemon id for teuthology.
IscsiServiceSpec(
service_type='iscsi',
service_id='iscsi',
),
DaemonDescription(
daemon_type='iscsi',
daemon_id="iscsi.a",
hostname="host1",
),
True
),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mgfritch a never ending story.

Comment on lines +643 to +736
# for cephadm, `service_id` is fixed to "iscsi" right now.
if self.service_id != 'iscsi':
raise ServiceSpecValidationError(
f'service_id must be `iscsi`, not `{self.service_id or "<unset>"}`')

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sebastian-philipp
Copy link
Contributor Author

@sebastian-philipp
Copy link
Contributor Author

sebastian-philipp commented Jun 17, 2020

we're getting there:

http://pulpito.ceph.com/swagner-2020-06-17_12:51:26-rados:cephadm-wip-swagner-testing-2020-06-17-1044-distro-basic-smithi/

NAME              HOST       STATUS         REFRESHED  AGE   VERSION               IMAGE NAME                                                     IMAGE ID      CONTAINER ID
alertmanager.a    smithi064  running (58s)  45s ago    87s   0.20.0                docker.io/prom/alertmanager:v0.20.0                            0881eb8f169f  57197cb9e95b
grafana.a         smithi110  running (68s)  43s ago    68s   6.6.2                 docker.io/ceph/ceph-grafana:latest                             87a51ecf0b1c  1f5ad971924f
iscsi.iscsi.a     smithi110  unknown        43s ago    2m    <unknown>             quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  <unknown>     <unknown>
mgr.x             smithi110  running (5m)   43s ago    5m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  34d66292a172
mon.a             smithi064  running (7m)   45s ago    7m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  dd95f663144b
mon.b             smithi110  running (5m)   43s ago    5m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  23c34627f336
mon.c             smithi064  running (5m)   45s ago    5m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  7e0b833920f9
node-exporter.a   smithi064  running (95s)  45s ago    99s   0.18.1                docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf  46ca0575b6df
node-exporter.b   smithi110  running (97s)  43s ago    100s  0.18.1                docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf  e13e6fca9855
osd.0             smithi064  running (4m)   45s ago    4m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  af5babcf31c8
osd.1             smithi064  running (4m)   45s ago    4m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  ec741b878300
osd.2             smithi064  running (3m)   45s ago    3m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  9892325c4c55
osd.3             smithi064  running (3m)   45s ago    3m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  f36dc5015dea
osd.4             smithi110  running (3m)   43s ago    3m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  e04147307f0c
osd.5             smithi110  running (2m)   43s ago    2m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  7f50e5a45f33
osd.6             smithi110  running (2m)   43s ago    2m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  ee99140f89ab
osd.7             smithi110  running (2m)   43s ago    2m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  2cd50c66685f
prometheus.a      smithi110  running (53s)  43s ago    103s  2.18.1                docker.io/prom/prometheus:v2.18.1                              de242295e225  7258e2816364
rgw.realm.zone.a  smithi064  running (2m)   45s ago    2m    16.0.0-2530-gb065994  quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504  84a05032a65c
NAME            RUNNING  REFRESHED  AGE   PLACEMENT                        IMAGE NAME                                                     IMAGE ID
alertmanager        1/1  48s ago    108s  count:1 smithi064=a              docker.io/prom/alertmanager:v0.20.0                            0881eb8f169f
grafana             1/1  46s ago    104s  count:1 smithi110=a              docker.io/ceph/ceph-grafana:latest                             87a51ecf0b1c
iscsi.iscsi         0/1  46s ago    2m    count:1 smithi110=iscsi.a        quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  <unknown>
mgr                 1/2  46s ago    5m    count:2 smithi110=x              quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504
mon.a               3/0  48s ago    -     <unmanaged>                      quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504
node-exporter       2/2  48s ago    117s  count:2 smithi064=a,smithi110=b  docker.io/prom/node-exporter:v0.18.1                           e5a616e4b9cf
osd.0               8/0  48s ago    -     <unmanaged>                      quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504
prometheus          1/1  46s ago    2m    count:1 smithi110=a              docker.io/prom/prometheus:v2.18.1                              de242295e225
rgw.realm.zone      1/1  48s ago    2m    count:1 smithi064=realm.zone.a   quay.io/ceph-ci/ceph:b065994edb71ac6ecb61030f0d12ee232255e4b8  41d676c08504
2020-06-17T13:22:12.608 INFO:ceph.iscsi.iscsi.a.smithi110.stdout:Jun 17 13:22:12 smithi110 bash[24044]: Error: relabel failed "/var/lib/ceph/6e7a2518-b09c-11ea-a06d-001a4aab830c/iscsi.iscsi.a/configfs": operation not supported
2020-06-17T13:22:12.613 INFO:ceph.iscsi.iscsi.a.smithi110.stdout:Jun 17 13:22:12 smithi110 systemd[1]: ceph-6e7a2518-b09c-11ea-a06d-001a4aab830c@iscsi.iscsi.a.service: main process exited, code=exited, status=127/n/a
2020-06-17T13:22:12.664 INFO:ceph.iscsi.iscsi.a.smithi110.stdout:Jun 17 13:22:12 smithi110 podman[24098]: Error: no container with name or ID ceph-6e7a2518-b09c-11ea-a06d-001a4aab830c-iscsi.iscsi.a found: no such container
2020-06-17T13:22:12.690 INFO:ceph.iscsi.iscsi.a.smithi110.stdout:Jun 17 13:22:12 smithi110 systemd[1]: Unit ceph-6e7a2518-b09c-11ea-a06d-001a4aab830c@iscsi.iscsi.a.service entered failed state.
2020-06-17T13:22:12.690 INFO:ceph.iscsi.iscsi.a.smithi110.stdout:Jun 17 13:22:12 smithi110 systemd[1]: ceph-6e7a2518-b09c-11ea-a06d-001a4aab830c@iscsi.iscsi.a.service failed.

@matthewoliver : Error: relabel failed "/var/lib/ceph/6e7a2518-b09c-11ea-a06d-001a4aab830c/iscsi.iscsi.a/configfs": operation not supported any idea?

@sebastian-philipp sebastian-philipp removed the wip-swagner-testing My Teuthology tests label Jun 18, 2020
@mgfritch
Copy link
Contributor

@matthewoliver : Error: relabel failed "/var/lib/ceph/6e7a2518-b09c-11ea-a06d-001a4aab830c/iscsi.iscsi.a/configfs": operation not supported any idea?

Needs #35141 ?

@matthewoliver
Copy link
Contributor

Oh it's doing a relabel on a virtual filesystem, that usually lives under /sys/kernel. what env is this? centos or something running selinux? I assume that's why a relabel is happening.

We may need to add a rule to src/selinux/. Let my pull up my selinux dev env again.

@matthewoliver
Copy link
Contributor

no matter how I attempt to restorecon in an attempt to relabel can I get the same error. It says it's coming from bash. Where can I find the script to see how they are relabeling?

@matthewoliver
Copy link
Contributor

maybe the relabel is a redherring, why is service failing and container not found?

@sebastian-philipp
Copy link
Contributor Author

here is the log of the failed run: http://pulpito.ceph.com/swagner-2020-06-17_12:51:26-rados:cephadm-wip-swagner-testing-2020-06-17-1044-distro-basic-smithi/5157644/ maybe you have more luck finding a possible hint?

@matthewoliver
Copy link
Contributor

Just occured to me, this is an ubuntu server. So maybe it's not actually selinux getting in the way but apparmor? Otherwise I'm confused about the "relabel" error message. "Relabel"ing is very selinux centric (I thought, though probably apparmor as well).

Is there a way to check and see if the ubuntu servers are in apparmor strcit (or enforce mode) as they say restrict mount access to configfs.

Or do I need to go down the apparmor/ubuntu rabbit hole next? :P

@matthewoliver
Copy link
Contributor

Hrmm, so I built up an Ubuntu server (vagrant) and the deployed a cephadm iscsi host, via vstart and it didn't seem to trigger anything in apparmor. I added more policies and still nothing.

I'll try and turn more debugging on just in case. However, maybe a vstart cluster doesn't install things in the same locations (which is why I tried deploying via cephadm because it mounts in /var/lib/ceph).
I didn't see a specific /var/lib rule in apparmor, though maybe it's related to the mount.

To see if there could be an apparmor policy hitting it, I wonder what sudo apparmor_status says on the ubuntu hosts. As I may just have enabled the policies wrong.

I'm new to apparmor, let me see if I can get some better logging happening.

@sebastian-philipp
Copy link
Contributor Author

let me first run this PR on more than one OS

@sebastian-philipp
Copy link
Contributor Author

let me first run this PR on more than one OS

http://pulpito.ceph.com/swagner-2020-06-24_11:51:16-rados:cephadm-wip-swagner2-testing-2020-06-24-1115-distro-basic-smithi/

(note, the green tests did fail as well, but were wrongly marked as pass)

e.g. on CentOS:

2020-06-24T12:23:43.578 INFO:ceph.iscsi.iscsi.a.smithi153.stdout:Jun 24 12:23:43 smithi153 podman[30317]: 2020-06-24 12:23:43.466341712 +0000 UTC m=+1.678687883 container remove 329798c1b8d3c22fb0d026a1c1fdd90394ef9f325ecb2d5a8847d8c04f622fd8 (image=quay.ceph.io
2020-06-24T12:23:43.578 INFO:ceph.iscsi.iscsi.a.smithi153.stdout:Jun 24 12:23:43 smithi153 bash[30262]: Error: relabel failed "/var/lib/ceph/874c2910-b614-11ea-a06d-001a4aab830c/iscsi.iscsi.a/configfs": operation not supported
2020-06-24T12:23:43.579 INFO:ceph.iscsi.iscsi.a.smithi153.stdout:Jun 24 12:23:43 smithi153 systemd[1]: ceph-874c2910-b614-11ea-a06d-001a4aab830c@iscsi.iscsi.a.service: Main process exited, code=exited, status=126/n/a
2020-06-24T12:23:43.579 INFO:ceph.iscsi.iscsi.a.smithi153.stdout:Jun 24 12:23:43 smithi153 systemd[1]: ceph-874c2910-b614-11ea-a06d-001a4aab830c@iscsi.iscsi.a.service: Failed with result 'exit-code'.

@gekios
Copy link
Contributor

gekios commented Jun 24, 2020

Hello, fyi I cherrypicked the commits from this PR and created a branch that is using my ceph-salt task so I can try to reproduce it downstream where it's more convenient since I can trigger a run with --sleep-before-teardown in order to connect there and check the logs/experiment with it to find what's the issue

@gekios
Copy link
Contributor

gekios commented Jul 3, 2020

hmm I ran the job downstream with sleep and even though didn't manage to reproduce the issue, the iscsi service container seems to be "flapping".

CONTAINER ID  IMAGE                                                                 COMMAND               CREATED                 STATUS                             PORTS  NAMES
a623276ed226  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -c %u %g /var/lib...  Less than a second ago  Exited (0) Less than a second ago         wizardly_chatterjee
4afb7ec3acf5  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  /bin/bash --login     6 seconds ago           Up 6 seconds ago                          ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-iscsi.a
f85a5e6f2ad8  docker.io/ceph/ceph-grafana:latest                                    /bin/bash             2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-grafana.a
b0b65899979d  docker.io/prom/node-exporter:v0.18.1                                  --no-collector.ti...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-node-exporter.b
c640156d8a7f  docker.io/prom/prometheus:v2.18.1                                     --config.file=/et...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-prometheus.a
044e3475b0e2  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.7 -f --set...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.7
93f483ff365f  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.6 -f --set...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.6
ddd09c99fff6  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.5 -f --set...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.5
05f51bb84fcd  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.4 -f --set...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.4
af3d31b8a2d6  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n mon.b -f --set...  2 hours ago             Up 2 hours ago                            ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-mon.b
target-geky-004:/home/ubuntu #    podman ps -a
CONTAINER ID  IMAGE                                                                 COMMAND               CREATED        STATUS            PORTS  NAMES
2ce2475002f5  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  /bin/bash --login     5 seconds ago  Up 4 seconds ago         ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-iscsi.a
f85a5e6f2ad8  docker.io/ceph/ceph-grafana:latest                                    /bin/bash             2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-grafana.a
b0b65899979d  docker.io/prom/node-exporter:v0.18.1                                  --no-collector.ti...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-node-exporter.b
c640156d8a7f  docker.io/prom/prometheus:v2.18.1                                     --config.file=/et...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-prometheus.a
044e3475b0e2  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.7 -f --set...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.7
93f483ff365f  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.6 -f --set...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.6
ddd09c99fff6  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.5 -f --set...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.5
05f51bb84fcd  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n osd.4 -f --set...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-osd.4
af3d31b8a2d6  registry.suse.de/devel/storage/7.0/containers/ses/7/ceph/ceph:latest  -n mon.b -f --set...  2 hours ago    Up 2 hours ago           ceph-efd152c0-bd18-11ea-86dc-fa163e22acf0-mon.b

@sebastian-philipp
Copy link
Contributor Author

relates to https://tracker.ceph.com/issues/46453

@sebastian-philipp
Copy link
Contributor Author

2020-09-10T12:41:25.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:24 smithi037 systemd[1]: ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service: Service RestartSec=10s expired, scheduling restart.
2020-09-10T12:41:25.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:24 smithi037 systemd[1]: ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service: Scheduled restart job, restart counter is at 1.
2020-09-10T12:41:25.152 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:24 smithi037 systemd[1]: Stopped Ceph iscsi.iscsi.a for 01f6039a-f362-11ea-922b-001a4aab830c.
2020-09-10T12:41:25.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:24 smithi037 systemd[1]: Starting Ceph iscsi.iscsi.a for 01f6039a-f362-11ea-922b-001a4aab830c...
2020-09-10T12:41:25.153 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 podman[15284]: a1bd4ec4b947b89bed48ddb0407ca89333635707430795003a3fa4f4685114a0
2020-09-10T12:41:25.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: Error: Failed to evict container: "": Failed to find container "ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a-tcmu" in state: no containe
2020-09-10T12:41:25.402 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a-tcmu
2020-09-10T12:41:25.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: Error: no container with ID or name "ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a-tcmu" found: no such container
2020-09-10T12:41:25.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: Error: Failed to evict container: "": Failed to find container "ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a" in state: no container wit
2020-09-10T12:41:25.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a
2020-09-10T12:41:25.403 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:25 smithi037 bash[15374]: Error: no container with ID or name "ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a" found: no such container
2020-09-10T12:41:26.651 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 bash[15374]: Error: relabel failed "/var/lib/ceph/01f6039a-f362-11ea-922b-001a4aab830c/iscsi.iscsi.a/configfs": failed to set file label on /var/lib/ceph/01f603
2020-09-10T12:41:26.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 bash[15374]: Error: relabel failed "/var/lib/ceph/01f6039a-f362-11ea-922b-001a4aab830c/iscsi.iscsi.a/configfs": failed to set file label on /var/lib/ceph/01f603
2020-09-10T12:41:26.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 systemd[1]: ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service: Control process exited, code=exited status=126
2020-09-10T12:41:26.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 bash[15774]: Error: no container with name or ID ceph-01f6039a-f362-11ea-922b-001a4aab830c-iscsi.iscsi.a-tcmu found: no such container
2020-09-10T12:41:26.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 systemd[1]: ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service: Failed with result 'exit-code'.
2020-09-10T12:41:26.652 INFO:journalctl@ceph.iscsi.iscsi.a.smithi037.stdout:Sep 10 12:41:26 smithi037 systemd[1]: Failed to start Ceph iscsi.iscsi.a for 01f6039a-f362-11ea-922b-001a4aab830c.

PLUS

2020-09-10 12:41:06,943 INFO Deploy daemon iscsi.iscsi.a ...
2020-09-10 12:41:06,943 DEBUG ['/bin/podman', 'run', '--rm', '--net=host', '--ipc=host', '-e', 'CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0', '-e', 'NODE_NAME=smithi037', '--entrypoint', 'stat', 'quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0', '-c', '%u %g', '/var/lib/ceph']
2020-09-10 12:41:06,943 DEBUG Running command: /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 -e NODE_NAME=smithi037 --entrypoint stat quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 -c %u %g /var/lib/ceph
2020-09-10 12:41:07,399 DEBUG --------------------------------------------------------------------------------
cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0', 'shell', '-c', '/etc/ceph/ceph.conf', '-k', '/etc/ceph/ceph.client.admin.keyring', '--fsid', '01f6039a-f362-11ea-922b-001a4aab830c', '--', 'ceph', 'orch', 'apply', 'prometheus', '1;smithi037=a']
2020-09-10 12:41:07,412 DEBUG Using specified fsid: 01f6039a-f362-11ea-922b-001a4aab830c
2020-09-10 12:41:07,412 DEBUG Using specified config: /etc/ceph/ceph.conf
2020-09-10 12:41:07,412 DEBUG ['/bin/podman', 'run', '--rm', '--net=host', '--ipc=host', '-e', 'CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0', '-e', 'NODE_NAME=smithi037', '--entrypoint', 'stat', 'quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0', '-c', '%u %g', '/var/lib/ceph']
2020-09-10 12:41:07,413 DEBUG Running command: /bin/podman run --rm --net=host --ipc=host -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 -e NODE_NAME=smithi037 --entrypoint stat quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 -c %u %g /var/lib/ceph
2020-09-10 12:41:07,695 DEBUG stat:stdout 167 167
2020-09-10 12:41:08,237 DEBUG stat:stdout 167 167
2020-09-10 12:41:08,634 INFO Creating ceph-iscsi config...
2020-09-10 12:41:08,635 INFO Write file: /var/lib/ceph/01f6039a-f362-11ea-922b-001a4aab830c/iscsi.iscsi.a/iscsi-gateway.cfg
2020-09-10 12:41:08,636 DEBUG Running command: systemctl daemon-reload
2020-09-10 12:41:08,735 DEBUG Running command: systemctl stop ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a
2020-09-10 12:41:08,763 DEBUG Running command (timeout=None): /bin/podman run --rm --net=host --ipc=host --privileged --group-add=disk -e CONTAINER_IMAGE=quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 -e NODE_NAME=smithi037 -v /var/run/ceph/01f6039a-f362-11ea-922b-001a4aab830c:/var/run/ceph:z -v /var/log/ceph/01f6039a-f362-11ea-922b-001a4aab830c:/var/log/ceph:z -v /var/lib/ceph/01f6039a-f362-11ea-922b-001a4aab830c/crash:/var/lib/ceph/crash:z -v /dev:/dev -v /run/udev:/run/udev -v /sys:/sys -v /run/lvm:/run/lvm -v /run/lock/lvm:/run/lock/lvm -v /etc/ceph/ceph.conf:/etc/ceph/ceph.conf:z -v /etc/ceph/ceph.client.admin.keyring:/etc/ceph/ceph.keyring:z --entrypoint ceph quay.ceph.io/ceph-ci/ceph:ef3557fd6d66214b1304b6c1c4004e6e2e4545a0 orch apply prometheus 1;smithi037=a
2020-09-10 12:41:08,767 DEBUG Running command: systemctl reset-failed ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a
2020-09-10 12:41:08,774 DEBUG systemctl:stderr Failed to reset failed state of unit ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service: Unit ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service not loaded.
2020-09-10 12:41:08,774 DEBUG Running command: systemctl enable ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a
2020-09-10 12:41:08,782 DEBUG systemctl:stderr Created symlink /etc/systemd/system/ceph-01f6039a-f362-11ea-922b-001a4aab830c.target.wants/ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a.service → /etc/systemd/system/ceph-01f6039a-f362-11ea-922b-001a4aab830c@.service.
2020-09-10 12:41:08,873 DEBUG Running command: systemctl start ceph-01f6039a-f362-11ea-922b-001a4aab830c@iscsi.iscsi.a

https://pulpito.ceph.com/swagner-2020-09-10_11:58:19-rados:cephadm-wip-swagner3-testing-2020-09-10-1058-distro-basic-smithi/5420635/

@sebastian-philipp
Copy link
Contributor Author

@sebastian-philipp
Copy link
Contributor Author

Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
Signed-off-by: Sebastian Wagner <sebastian.wagner@suse.com>
@sebastian-philipp
Copy link
Contributor Author

rebased

@sebastian-philipp
Copy link
Contributor Author

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants