qa/tasks/cephfs/mount.py: defer deleting the netnses and bridge by lxbsz · Pull Request #35944 · ceph/ceph

lxbsz · 2020-07-06T12:32:54Z

Fixes: https://tracker.ceph.com/issues/46282
Signed-off-by: Xiubo Li xiubli@redhat.com

Checklist

References tracker ticket
Updates documentation if necessary
Includes tests for new functionality or reproducer for bug

Show available Jenkins commands

jenkins retest this please
jenkins test classic perf
jenkins test crimson perf
jenkins test signed
jenkins test make check
jenkins test make check arm64
jenkins test submodules
jenkins test dashboard
jenkins test dashboard backend
jenkins test docs
jenkins render docs
jenkins test ceph-volume all
jenkins test ceph-volume tox

qa/tasks/cephfs/mount.py

lxbsz · 2020-07-09T01:37:15Z

jenkins retest this please

batrick · 2020-07-09T02:21:44Z

qa/tasks/cephfs/mount.py

-        args = ["sudo", "bash", "-c",
-                "iptables -A FORWARD -o {0} -i ceph-brx -j ACCEPT".format(gw)]
+        args = ['sudo', 'iptables', '-A', 'FORWARD', '-o', '{0}'.format(gw), '-i', 'ceph-brx', '-j', 'ACCEPT']
        self.client_remote.run(args=args, timeout=(5*60), omit_sudo=False)


While you're here fixing this, why not do:

self.run_shell_payload(f""" sudo iptables ... sudo iptables ... """, omit_sudo=False)

Cool, I didn't notice this helper, will fix it.
Thanks.

lxbsz · 2020-07-10T01:01:52Z

jenkins retest this please

batrick

flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/mount.py:279:0: F541 f-string is missing placeholders
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/flake8/bin/flake8 --select=F,E9 --exclude=venv,.tox (exited with code 1)

make check failure ^

batrick

Otherwise LGTM.

batrick · 2020-07-11T00:02:19Z

qa/tasks/cephfs/mount.py


+# This will cleanup the stale netnses, which are from the
+# last failed test cases.
+def cleanup_stale_netnses_and_bridge(remote):


Let's make this a static method of CephFSMount.

Sure, done.

lxbsz · 2020-07-13T01:31:44Z

flake8 run-test: commands[0] | flake8 --select=F,E9 --exclude=venv,.tox
./tasks/cephfs/mount.py:279:0: F541 f-string is missing placeholders
ERROR: InvocationError for command /home/jenkins-build/build/workspace/ceph-pull-requests/qa/.tox/flake8/bin/flake8 --select=F,E9 --exclude=venv,.tox (exited with code 1)

make check failure ^

Fixed it. Thx.

lxbsz · 2020-07-13T06:26:43Z

jenkins test dashboard backend

Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz · 2020-07-16T05:56:57Z

Without setting omit_sudo to False, these commands will have no effect.

Done. thanks.

batrick · 2020-07-16T14:07:58Z

jenkins retest this please

qa/tasks/cephfs/mount.py

lxbsz · 2020-07-17T03:23:07Z

jenkins test dashboard backend

batrick · 2020-07-28T02:30:59Z

in 2h: https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20200728.022107

batrick

Found this error in QA:

2020-07-28T04:11:10.187 INFO:teuthology.orchestra.run.smithi094:> (cd /home/ubuntu/cephtest/mnt.0 && exec sudo bash -c '
2020-07-28T04:11:10.188 INFO:teuthology.orchestra.run.smithi094:>                 set -e
2020-07-28T04:11:10.188 INFO:teuthology.orchestra.run.smithi094:>                 sudo ip link add name ceph-brx type bridge
2020-07-28T04:11:10.189 INFO:teuthology.orchestra.run.smithi094:>                 sudo ip addr flush dev ceph-brx
2020-07-28T04:11:10.189 INFO:teuthology.orchestra.run.smithi094:>                 sudo ip link set ceph-brx up
2020-07-28T04:11:10.190 INFO:teuthology.orchestra.run.smithi094:>                 sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx
2020-07-28T04:11:10.191 INFO:teuthology.orchestra.run.smithi094:>             ')
2020-07-28T04:11:10.232 INFO:teuthology.orchestra.run.smithi094.stderr:bash: line 0: cd: /home/ubuntu/cephtest/mnt.0: No such file or directory

From: /ceph/teuthology-archive/pdonnell-2020-07-28_03:46:25-fs-wip-pdonnell-testing-20200728.022107-distro-basic-smithi/5262950/teuthology.log

Xiubo, please run through teuthology when you've updated your PR so we can get this ready to merge ASAP.

batrick · 2020-07-28T16:11:39Z

qa/tasks/cephfs/mount.py

+                sudo ip addr flush dev ceph-brx
+                sudo ip link set ceph-brx up
+                sudo ip addr add {ip}/{mask} brd {brd} dev ceph-brx
+            """, timeout=(5*60), omit_sudo=False)


Suggested change

""", timeout=(5*60), omit_sudo=False)

""", timeout=(5*60), omit_sudo=False, cwd='/')

batrick · 2020-07-28T16:11:59Z

qa/tasks/cephfs/mount.py

+            sudo iptables -A FORWARD -o {gw} -i ceph-brx -j ACCEPT
+            sudo iptables -A FORWARD -i {gw} -o ceph-brx -j ACCEPT
+            sudo iptables -t nat -A POSTROUTING -s {ip}/{mask} -o {gw} -j MASQUERADE
+        """, timeout=(5*60), omit_sudo=False)


Suggested change

""", timeout=(5*60), omit_sudo=False)

""", timeout=(5*60), omit_sudo=False, cwd='/')

batrick · 2020-07-28T16:12:15Z

qa/tasks/cephfs/mount.py

+                set -e
+                sudo ip netns add {self.netns_name}
+                sudo ip netns set {self.netns_name} {nsid}
+            """, timeout=(5*60), omit_sudo=False)


Suggested change

""", timeout=(5*60), omit_sudo=False)

""", timeout=(5*60), omit_sudo=False, cwd='/')

batrick · 2020-07-28T16:12:27Z

qa/tasks/cephfs/mount.py

+            sudo ip netns exec {self.netns_name} ip link set veth0 up
+            sudo ip netns exec {self.netns_name} ip link set lo up
+            sudo ip netns exec {self.netns_name} ip route add default via {brxip}
+        """, timeout=(5*60), omit_sudo=False)


batrick · 2020-07-28T16:12:32Z

qa/tasks/cephfs/mount.py

+            set -e
+            sudo ip link set brx.{nsid} up
+            sudo ip link set dev brx.{nsid} master ceph-brx
+        """, timeout=(5*60), omit_sudo=False)


batrick · 2020-07-28T16:12:39Z

qa/tasks/cephfs/mount.py

+            sudo ip link set brx.{self.nsid} down
+            sudo ip link delete dev brx.{self.nsid}
+            sudo ip netns delete {self.netns_name}
+        """, timeout=(5*60), omit_sudo=False)


batrick · 2020-07-28T16:12:45Z

qa/tasks/cephfs/mount.py

+            set -e
+            sudo ip link set ceph-brx down
+            sudo ip link delete ceph-brx
+        """, timeout=(5*60), omit_sudo=False)


batrick · 2020-07-28T16:12:56Z

qa/tasks/cephfs/mount.py

+            sudo iptables -D FORWARD -o {gw} -i ceph-brx -j ACCEPT
+            sudo iptables -D FORWARD -i {gw} -o ceph-brx -j ACCEPT
+            sudo iptables -t nat -D POSTROUTING -s {ip}/{mask} -o {gw} -j MASQUERADE
+        """, timeout=(5*60), omit_sudo=False)


batrick · 2020-07-28T16:14:27Z

Xiubo, please run through teuthology when you've updated your PR so we can get this ready to merge ASAP.

Here's what I used ot test:

teuthology-suite -m smithi -p 29 --suite fs --ceph wip-pdonnell-testing-20200728.022107 --filter client-recovery

Signed-off-by: Xiubo Li <xiubli@redhat.com>

Once we have run the test cases and the ceph-brx bridge is setup, it will save the config in "/etc/sysconfig/network-scripts/ifcfg-ceph-brx" or somewhere else. It will be kept after the ceph-brx bridge removed. So next time once the ceph-brx bridge is created or added, it will read the config from it, then when we config it again we will get error like: "RTNETLINK answers: File exists" Here we need to flush it before config it. Fixes: https://tracker.ceph.com/issues/45817 Signed-off-by: Xiubo Li <xiubli@redhat.com>

If the previous test cases failed, the netnses and bridge will be left. Here will remove them when new test cases begin. Fixes: https://tracker.ceph.com/issues/45806 Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz · 2020-07-29T01:01:38Z

Xiubo, please run through teuthology when you've updated your PR so we can get this ready to merge ASAP.

Here's what I used ot test:

teuthology-suite -m smithi -p 29 --suite fs --ceph wip-pdonnell-testing-20200728.022107 --filter client-recovery

Sure, fixed them all and I am now running the test, the branch is wip-lxbsz-testing-20200729-0855.

Thanks.

lxbsz · 2020-07-29T01:53:48Z

jenkins test dashboard backend

lxbsz · 2020-07-30T00:56:10Z

Updated it with a small fix, for some test cases they will call mount_a.kill() first, which will suspend the netns by bring the network interface down, and a while later they will call the kill_cleanup() to do the unmount, if we reuse the netns later in the next test case, the netns will keep suspended, so we need to resume it.

The netnses maybe created/deleted many times in the whole test cases, we can defer cleaning them untile the last mountpoint is unmounted or when the test is exiting. Fixes: https://tracker.ceph.com/issues/46282 Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz · 2020-07-30T07:23:51Z

Xiubo, please run through teuthology when you've updated your PR so we can get this ready to merge ASAP.

Here's what I used ot test:

teuthology-suite -m smithi -p 29 --suite fs --ceph wip-pdonnell-testing-20200728.022107 --filter client-recovery

Test done and passed, please see: https://pulpito.ceph.com/xiubli-2020-07-30_04:53:06-fs-wip-lxbsz-testing-20200730-0901-distro-basic-smithi/

batrick · 2020-07-31T21:12:28Z

https://pulpito.ceph.com/?branch=wip-pdonnell-testing-20200730.193838

Failures known/unrelated.

lxbsz requested review from batrick, jtlayton and rishabh-d-dave July 6, 2020 12:33

lxbsz force-pushed the client_delay branch from 91d2012 to 95aab44 Compare July 6, 2020 12:36

jtlayton reviewed Jul 6, 2020

View reviewed changes

qa/tasks/cephfs/mount.py Outdated Show resolved Hide resolved

batrick added cephfs Ceph File System needs-review labels Jul 6, 2020

lxbsz force-pushed the client_delay branch 6 times, most recently from 3a2d705 to 6793049 Compare July 8, 2020 13:01

batrick requested changes Jul 9, 2020

View reviewed changes

lxbsz force-pushed the client_delay branch from 6793049 to 4419acc Compare July 9, 2020 06:33

lxbsz changed the title ~~qa/tasks/cephfs/mount.py: delete veth pair instead of single peer int…~~ qa/tasks/cephfs/mount.py: defer deleting the netnses and bridge Jul 9, 2020

batrick requested changes Jul 11, 2020

View reviewed changes

lxbsz force-pushed the client_delay branch 2 times, most recently from 5f075a3 to 0af0bf7 Compare July 13, 2020 01:29

lxbsz requested a review from batrick July 13, 2020 01:31

lxbsz force-pushed the client_delay branch from 0af0bf7 to b1a143b Compare July 13, 2020 04:41

qa/tasks/cephfs/mount.py: clean up the none used code

2524af9

Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz force-pushed the client_delay branch from b1a143b to fad9edd Compare July 14, 2020 10:28

batrick approved these changes Jul 16, 2020

View reviewed changes

batrick removed the needs-review label Jul 16, 2020

lxbsz force-pushed the client_delay branch from fad9edd to e44cdb5 Compare July 16, 2020 05:55

lxbsz mentioned this pull request Jul 16, 2020

qa/tasks/vstart_runner.py: remove the stale netnses and bridge #35283

Closed

3 tasks

rishabh-d-dave approved these changes Jul 16, 2020

View reviewed changes

batrick requested changes Jul 16, 2020

View reviewed changes

qa/tasks/cephfs/mount.py Outdated Show resolved Hide resolved

batrick removed needs-qa wip-pdonnell-testing labels Jul 16, 2020

lxbsz force-pushed the client_delay branch 2 times, most recently from f1e2b98 to 5c3db71 Compare July 17, 2020 01:18

batrick requested changes Jul 28, 2020

View reviewed changes

lxbsz added 3 commits July 29, 2020 08:40

qa/tasks/cephfs/mount.py: switch to run_shell_payload() helper

f8d8e30

Signed-off-by: Xiubo Li <xiubli@redhat.com>

qa/tasks/cephfs/mount.py: remove the stale netnses and bridge

c0c5550

If the previous test cases failed, the netnses and bridge will be left. Here will remove them when new test cases begin. Fixes: https://tracker.ceph.com/issues/45806 Signed-off-by: Xiubo Li <xiubli@redhat.com>

lxbsz force-pushed the client_delay branch from 5c3db71 to 3e5b78a Compare July 29, 2020 00:59

lxbsz force-pushed the client_delay branch from 3e5b78a to ae38dda Compare July 30, 2020 00:53

lxbsz force-pushed the client_delay branch from ae38dda to 4760002 Compare July 30, 2020 00:58

batrick approved these changes Jul 30, 2020

View reviewed changes

batrick added needs-qa wip-pdonnell-testing labels Jul 30, 2020

batrick merged commit ca7be68 into ceph:master Jul 31, 2020

	""", timeout=(5*60), omit_sudo=False)
	""", timeout=(5*60), omit_sudo=False, cwd='/')

Conversation

lxbsz commented Jul 6, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Checklist

Uh oh!

Uh oh!

lxbsz commented Jul 9, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxbsz commented Jul 10, 2020

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

lxbsz commented Jul 13, 2020

Uh oh!

lxbsz commented Jul 13, 2020

Uh oh!

lxbsz commented Jul 16, 2020

Uh oh!

batrick commented Jul 16, 2020

Uh oh!

Uh oh!

lxbsz commented Jul 17, 2020

Uh oh!

batrick commented Jul 28, 2020

Uh oh!

batrick left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

batrick commented Jul 28, 2020

Uh oh!

lxbsz commented Jul 29, 2020

Uh oh!

lxbsz commented Jul 29, 2020

Uh oh!

lxbsz commented Jul 30, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

lxbsz commented Jul 30, 2020

Uh oh!

batrick commented Jul 31, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

lxbsz commented Jul 6, 2020 •

edited

Loading

lxbsz commented Jul 30, 2020 •

edited

Loading