virtcontainers: netmon: Monitor network changes by sboeuf · Pull Request #534 · kata-containers/runtime

sboeuf · 2018-07-30T18:27:25Z

This commit introduces a new watcher dedicated to the monitoring
of a specific network namespace in order to detect any change that
could happen to the network.

As a result of such a detection, the watcher should call into the
appropriate runtime path with the correct arguments to modify the
pod network accordingly.

Fixes #170

Signed-off-by: Sebastien Boeuf sebastien.boeuf@intel.com

sboeuf · 2018-07-30T18:29:26Z

This PR supersedes the original C implementation #194

devimc · 2018-07-30T18:29:57Z

virtcontainers/netmon/netmon.go

+
+	netlinkFamily = netlink.FAMILY_V4
+
+	storageParentPath = "/var/run/kata-containers/netmon/sbs"


this path should be generated by in the Makefile

Good point.

devimc · 2018-07-30T18:31:05Z

virtcontainers/netmon/netmon.go

+		os.Exit(1)
+	}
+
+	/*


why is this commented? is this a TODO ?

Oh yeah, I forgot to remove it. I'll do that when I'll update the PR.

devimc · 2018-07-30T18:33:04Z

virtcontainers/netmon/netmon.go

+		if addr.IPNet == nil {
+			continue
+		}
+	}


and this loop if for ... ?

Oh yeah interesting I forgot to complete the implementation there !

sboeuf · 2018-07-30T18:39:50Z

Things left to do:

Uses logrus package for proper logging
Add a Makefile
Add code to virtcontainers:
- Spawn this binary inside the netns after the netns has been created
- Change the internal network interfaces name (tap, bridges) by adding "_kata" suffix.
Improve use case for crash/reboot (save/restore from filesystem)

katacontainersbot · 2018-07-30T18:46:09Z

PSS Measurement:
Qemu: 169758 KB
Proxy: 5822 KB
Shim: 9154 KB

Memory inside container:
Total Memory: 2043480 KB
Free Memory: 2003704 KB

katacontainersbot · 2018-07-30T19:48:22Z

PSS Measurement:
Qemu: 167586 KB
Proxy: 5746 KB
Shim: 8928 KB

Memory inside container:
Total Memory: 2043480 KB
Free Memory: 2003564 KB

opendev-zuul · 2018-07-30T20:16:55Z

Build succeeded (third-party-check pipeline).

kata-runsh : SUCCESS in 57m 32s

jodh-intel · 2018-07-31T10:01:12Z

virtcontainers/netmon/netmon.go

+
+const (
+	netmonName    = "kata-netmon"
+	netmonVersion = "0.0.1"


These two variables should probably be set by the build (particularly version) as we do for the runtime itself.

Yes they could be generated through the build.

jodh-intel · 2018-07-31T10:02:06Z

virtcontainers/netmon/netmon.go

+
+	kataSuffix = "kata"
+
+	netlinkFamily = netlink.FAMILY_V4


Worth mentioning anything about IPv6 here?

Yes I'll add a comment that we want to focus on ipv4 for now, just to make sure we're not introducing too much complexity at the first step.

jodh-intel · 2018-07-31T10:04:23Z

virtcontainers/netmon/netmon.go

+}
+
+func printVersion() {
+	fmt.Printf("%s version %s\n", netmonName, netmonVersion)


This is a new system component, so please could you update kata-env to display details of it (including running kata-netmon --version as we do for other components).

Sure, but this will come later, I need to get the code working and tested first.

jodh-intel · 2018-07-31T10:10:14Z

virtcontainers/netmon/netmon.go

+	flag.BoolVar(&params.debug, "d", false, "enable debug mode")
+	flag.BoolVar(&version, "v", false, "display program version and exit")
+	flag.StringVar(&params.sandboxID, "s", "", "sandbox id (required)")
+	flag.StringVar(&params.runtimePath, "r", "", "runtime path (required)")


flag auto-generates a --help option but it would be useful to override the default to display a description of the program too imho.

As mentioned on the original C-based PR, wouldn't it be better to call the vc API rather than having the overhead of launching instances of the runtime (even if they are short-lived)?

flag auto-generates a --help option but it would be useful to override the default to display a description of the program too imho.

Do you mean the program usage ? Because that's the way it works by default. I think this improvement can definitely be part of a follow up PR.

As mentioned on the original C-based PR, wouldn't it be better to call the vc API rather than having the overhead of launching instances of the runtime (even if they are short-lived)?

Well we could do that from the Go implementation, but what if we move to C implementation later ? This is not gonna be pretty to provide the Kata API as a C library...
Also, by doing so, we pull the entire dependencies from Kata by importing the API as a package, meaning we'll end up with a pretty huge binary running on the system.
Technically, this is doable, but I don't know if we want to go this way.

IMO, one benefit of calling the kata API directly is that if you modularize it properly, we can create a goroutine to monitor the netns in containerd kata shim and that presents no dependency on the kata-runtime cli, which is good from containerd kata shim perspective.

How about a structure like this:

virtcontainers/netnsmon: a package that calls kata API to do netns monitoring

cli/netmon: a command line built on top of virtcontainers/netnsmon. Other kata-runtime sub-command can invoke it to create a daemon to monitor the netns

for containerd kata shim, it can create a goroutine that calls virtcontainers/netnsmon to monitor the netns when it wants to.

In both kata-runtime and containerd kata shim case, virtcontainers do not invoke netnsmon but let the callers decide and drive the procedure.

WDTY?

cc @gnawux @lifupan

@bergwolf
If we decide to go this way, I would appreciate if this could come as a follow up PR, since I have reworked this PR quite a few times now, and it's pretty big, and I would like to see it landing at some point.

jodh-intel · 2018-07-31T10:11:43Z

virtcontainers/netmon/netmon.go

+
+		src := ""
+		if netRoute.Src.To4() != nil {
+			dst = netRoute.Src.String()


You mean src = right?

Thanks for noticing ;)

jodh-intel · 2018-07-31T10:17:30Z

virtcontainers/netmon/netmon.go

+	}
+}
+
+func main() {


Please could you add the standard signal and crash handing code. To avoid duplicating code inside this repo, you could do what I did in the throttler repo and create a new signals package:

https://github.com/kata-containers/ksm-throttler/blob/master/pkg/signals/signals.go

Then:

func main() { defer ksig.HandlePanic() realMain() }

And then add calls to:

set CrashOnErorr behaviour.

Call that packages SetLogger().

Create the signal-handling loop with calls to HandledSignals(), FatalSignal() and NonFatalSignal().

That sounds like a nice thing to do ;)

jodh-intel · 2018-07-31T10:20:05Z

virtcontainers/netmon/netmon.go

+
+	storageParentPath = "/var/run/kata-containers/netmon/sbs"
+	storageDirPerm    = 0750
+	sharedFile        = "shared.json"


Please document what this is for.

katacontainersbot · 2018-07-31T21:52:15Z

PSS Measurement:
Qemu: 165604 KB
Proxy: 5872 KB
Shim: 8774 KB

Memory inside container:
Total Memory: 2043480 KB
Free Memory: 2003936 KB

katacontainersbot · 2018-07-31T22:41:20Z

PSS Measurement:
Qemu: 167995 KB
Proxy: 5830 KB
Shim: 9198 KB

Memory inside container:
Total Memory: 2043480 KB
Free Memory: 2003332 KB

opendev-zuul · 2018-07-31T23:22:30Z

Build succeeded (third-party-check pipeline).

kata-runsh : SUCCESS in 51m 40s

bergwolf · 2018-08-01T04:33:43Z

To clarify, this depends on #287 for the kata cli network sub-command. Adding WiP because of it.

sboeuf · 2018-08-01T05:45:23Z

Thanks @bergwolf ! Indeed, it relies on the fact that #287 needs to be merged first.

katacontainersbot · 2018-08-01T06:00:20Z

PSS Measurement:
Qemu: 165854 KB
Proxy: 5851 KB
Shim: 8934 KB

Memory inside container:
Total Memory: 2043480 KB
Free Memory: 2003892 KB

codecov · 2018-08-01T06:39:25Z

Codecov Report

Merging #534 into master will decrease coverage by 0.14%.
The diff coverage is 60.64%.

@@            Coverage Diff             @@
##           master     #534      +/-   ##
==========================================
- Coverage   65.25%   65.11%   -0.15%     
==========================================
  Files          85       87       +2     
  Lines        9978    10373     +395     
==========================================
+ Hits         6511     6754     +243     
- Misses       2815     2942     +127     
- Partials      652      677      +25

opendev-zuul · 2018-08-01T06:45:28Z

Build succeeded (third-party-check pipeline).

kata-runsh : SUCCESS in 57m 50s

ghost · 2018-08-29T09:56:32Z

@sboeuf #287 has been merged now but since opentracing support is being added for kata as tracked here , Do we wish to keep this PR open? considering its gonna be unnecessary after opentracing is added.

jodh-intel · 2018-08-29T10:32:51Z

Hi @ydjainopensource - I think we'll still need this as the tracing can only show that changes occurred - this new application needs to modify the network.

ghost · 2018-08-29T10:33:40Z

Okay no issues we will keep this open

ghost · 2018-09-04T02:55:26Z

I have tested the commit base on branch master after add some log print, but there is a little strange problem: call kata-runtime kata-netwowk add-iface in netmon will fail。
$ sudo docker run --rm --name test -tid alpine sleep 10000

023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0ad4500e

$ sudo nsenter --net=$(sudo docker inspect test |jq -r .[0].NetworkSettings.SandboxKey) ./kata-netmon -d -log info -r /usr/bin/kata-runtime -s 023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0ad4500e

I will get error messages after sudo docker network connect newnetwork test:

INFO[0062] Exec command: &{Path:/usr/bin/kata-runtime Args:[/usr/bin/kata-runtime kata-network add-iface 023313cf56b58d3283fefc84f4d1787cac2916e758a1d190
12c7ff6a0ad4500e /var/run/kata-containers/netmon/sbs/023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0ad4500e/shared.json] Env:[] Dir: Stdin: Std
out: Stderr: ExtraFiles:[] SysProcAttr: Process: ProcessState: ctx: lookPathErr: finished:false childFiles:[] closeAfterSta
rt:[] closeAfterWait:[] goroutine:[] errch: waitDone:} name=kata-netmon pid=22782 sandbox=023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0
ad4500e source=netmon
FATA[0062] handleEvents() error="exit status 1" name=kata-netmon pid=22782 sandbox=023313cf56b58d3283fefc84f4d1787cac2916e758a1
d19012c7ff6a0ad4500e source=netmon

but it will work if execute the command manually:
sudo nsenter --net=$(sudo docker inspect test |jq -r .[0].NetworkSettings.SandboxKey) /usr/bin/kata-runtime kata-network add-iface 023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0ad4500e /var/run/kata-containers/netmon/sbs/023313cf56b58d3283fefc84f4d1787cac2916e758a1d19012c7ff6a0ad4500e/shared.json

and the new tap tap1 interface created Successfully

$ sudo nsenter --net=$(sudo docker inspect test |jq -r .[0].NetworkSettings.SandboxKey) ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
2: ip_vti0@NONE: mtu 1332 qdisc noop state DOWN group default qlen 1
link/ipip 0.0.0.0 brd 0.0.0.0
9: eth0@if10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:00:ca:fe:00:00 brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: eth1@if12: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:00:ca:fe:00:01 brd ff:ff:ff:ff:ff:ff link-netnsid 0
47124: tap1@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1500
link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff
44374: tap0@eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1500
link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff

so maybe there's some permissions issue in netmon or the way of my use is wrong?

sboeuf · 2018-09-04T17:17:22Z

@ningbo9 the PR is not completely ready as it does not include the modification inside kata-runtime itself to launch the network monitor.

bergwolf · 2018-09-13T07:43:04Z

cli/config/configuration.toml.in

+# network being added to the existing network namespace, after the
+# sandbox has been created.
+# (default: disabled)
+#enable = true


nits: as I put a test commit at kata-containers/tests#743, I noticed that it would be better to call this enable_netmon instead, so that we can easily grep and replace it in scripts.

Of course I can do that and I agree this might help!

bergwolf · 2018-09-13T07:47:47Z

btw, @sboeuf if you want to merge this PR feel free to do so. It would be great if you can rename the enable option to enable_netmon though.

Anyway, whatever error I'm seeing w/ network hotplug, it is not related to your PR. So I'm giving my
LGTM

caoruidong · 2018-09-13T09:21:33Z

Maybe the different guest rootfs distro or kernel causes this error ?

bergwolf · 2018-09-13T10:18:50Z

@caoruidong I've tried alpine based initrd w/ agent as init, and centos based rootfs image w/ systemd as init, both failed the same way. It seems kata-containers/tests#743 have given the same results:

+ sudo docker network create foobartestnetwork
4485b0308d12816eabc6ffbb900228c12b349dee3866aa1211ec1d946c5c0511
+ sudo docker run --runtime=kata-runtime -d --name kata-vKBoFN busybox tail -f /dev/null
b3c68f2d9c37de4176f1de711cc43b090632c744d95f17196643c0fc983dac33
+ sudo docker network connect foobartestnetwork kata-vKBoFN
+ sleep 10
++ sudo docker exec -it kata-vKBoFN ip addr show eth1
++ sed -ne 's/.*inet \([.0-9]\{7,15\}\)\/[0-9]\{1,2\} .*/\1/p'
the input device is not a TTY
+ ipaddr=
+ sudo docker rm -f kata-vKBoFN
kata-vKBoFN
+ sudo docker network rm foobartestnetwork
foobartestnetwork
+ echo hotplugged network ip address is
hotplugged network ip address is
+ test -n ''

@sboeuf what guest rootfs distro are you testing with?

caoruidong

LGTM

caoruidong · 2018-09-13T14:34:55Z

netmon/netmon.go

+type Interface struct {
+	Device      string       `json:"device,omitempty"`
+	Name        string       `json:"name,omitempty"`
+	IPAddresses []*IPAddress `json:"IPAddresses,omitempty"`


Capitalization? OK.

Oh! You mean from the json definition?
Yes I'll change that!

Yes. Thanks.

Oh wait, I did that so that it comply with the structure defined in agent/protocols/grpc/. I'm afraid the unmarshalling might not work if I change this.

Fair enough!

katacontainersbot · 2018-09-13T16:00:51Z

PSS Measurement:
Qemu: 165597 KB
Proxy: 4063 KB
Shim: 9015 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2006712 KB

opendev-zuul · 2018-09-13T16:45:53Z

Build failed (third-party-check pipeline) integration testing with
OpenStack. For information on how to proceed, see
http://docs.openstack.org/infra/manual/developers.html#automated-testing

kata-runsh : TIMED_OUT in 1h 01m 07s

sboeuf · 2018-09-13T19:07:46Z

@amshinde PTAL and feel free to merge if you're fine with it.

amshinde

lgtm @sboeuf
Just some minor nits.

amshinde · 2018-09-13T23:47:11Z

netmon/netmon.go

+	// For simplicity the code will only focus on IPv4 addresses for now.
+	netlinkFamily = netlink.FAMILY_V4
+
+	storageParentPath = "/var/run/kata-containers/netmon/sbs"


Should this go in the Makefile?

The Makefile does not contain any other storage path yet. Should this be part of a more global PR pushing the default storage paths to the Makefile?
WDYT?

Ok lets address in a separate PR.

amshinde · 2018-09-13T23:59:01Z

netmon/netmon.go

+	}
+
+	// First, ignore if the interface name contains "kata". This way we
+	// are preventing from adding interfaces created by Kata Containers.


/adding/deleting/

amshinde · 2018-09-14T00:12:05Z

netmon/netmon.go

+
+	// First, ignore if the interface name contains "kata". This way we
+	// are preventing from adding interfaces created by Kata Containers.
+	if strings.Contains(linkAttrs.Name, kataSuffix) {


Why not just use strings.HasSuffix here?

Let's see what we can get in CI environment. Please DO NOT MERGE! Depends-on: github.com/kata-containers/runtime#534 Fixes: #999999999 Signed-off-by: Peng Tao <bergwolf@gmail.com>

This commit introduces a new watcher dedicated to the monitoring of a specific network namespace in order to detect any change that could happen to the network. As a result of such a detection, the watcher should call into the appropriate runtime path with the correct arguments to modify the pod network accordingly. Fixes kata-containers#170 Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

In order to reduce the overhead due to the import of the whole agent protocol, only the needed structures are duplicated. This is a temporary solution, and those structures should be defined into their own package to prevent from such overhead. Note: the overhead of the binray size went down from 15MiB to 3MiB when this commit removed the dependency on the agent protocol. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

Instead of dumping logs through the standard output with fmt.Printf() function, this commit improves the logging by relying on logrus. Also, it relies on the syslog hook so that all the logs get redirected to the journal. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

This commit modifies the Makefile at the root of this repository so that the binary kata-netmon can be built from there. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

This commit adds some unit testing in order to validate some of the new code that have been introduced with the new network monitor. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

8abc400 agent: add test to WaitProcess() f746ed8 agent: allow multiple waitProcess() 157f1c1 travis: Add variable needed to run static checks ed54087 travis: bump golang version ba0c7fc client: wait for session to be fully closed 0865c98 agent: wait session to be fully shutdown 55f1480 vendor: update yamux dependency 5e36bfc network: Wait for network device in UpdateInterface 218ce89 device: Rename getBlockDeviceNodeName to getPCIDeviceName c9a4e2e uevent: Store the interface field as device name for network interfaces 74a5364 build: fix make proto error b1c2ad8 agent: add support for online memory and cpu separately. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

Because the network monitor will be listening to every event received through the netlink socket, it will be notified everytime a new link will be added/updated/modified in the network namespace it's running into. The goal being to detect new interface added by Docker such as a veth pair. The problem is that kata-runtime will add other internal interfaces when the network monitor will ask for the addition of the new veth pair. And we need a way to ignore those new interfaces being created as they relate to the veth pair that is being added. That's why, in order to prevent from running into an infinite loop, virtcontainers needs to tag the internal interfaces with the "kata" suffix so that the network monitor will be able to ignore them. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

This patch enables the code responsible for starting and stopping the network monitor. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

In order to choose if the network monitor should be used or not, this patch makes it configurable from the configuration.toml file. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

sboeuf · 2018-09-14T16:39:17Z

@amshinde changes applied, PTAL.

amshinde · 2018-09-14T17:47:17Z

lgtm.
@sboeuf CI seems to be failing on ubuntu.

sboeuf · 2018-09-14T17:48:46Z

@amshinde yeah I've just restarted because three builds failed because of networking issues. Nothing related to the PR.

katacontainersbot · 2018-09-14T17:53:28Z

PSS Measurement:
Qemu: 169337 KB
Proxy: 4017 KB
Shim: 8739 KB

Memory inside container:
Total Memory: 2043460 KB
Free Memory: 2006952 KB

sboeuf · 2018-09-14T19:34:27Z

@amshinde could you please merge this one?

sboeuf mentioned this pull request Jul 30, 2018

[WIP] virtcontainers: netns_watcher: Monitor network changes #194

Closed

devimc reviewed Jul 30, 2018

View reviewed changes

sboeuf force-pushed the monitor_network_golang branch from a890338 to 42ebbd4 Compare July 30, 2018 19:03

jodh-intel reviewed Jul 31, 2018

View reviewed changes

sboeuf force-pushed the monitor_network_golang branch 2 times, most recently from a60e56e to 849e97e Compare July 31, 2018 21:29

sboeuf force-pushed the monitor_network_golang branch from 849e97e to 80b905e Compare July 31, 2018 22:27

bergwolf added the wip label Aug 1, 2018

sboeuf force-pushed the monitor_network_golang branch from 80b905e to 3124c9d Compare August 1, 2018 05:44

jodh-intel mentioned this pull request Aug 1, 2018

Adding OpenTracing support for Kata kata-containers/kata-containers#27

Closed

bergwolf reviewed Sep 13, 2018

View reviewed changes

caoruidong approved these changes Sep 13, 2018

View reviewed changes

caoruidong reviewed Sep 13, 2018

View reviewed changes

sboeuf force-pushed the monitor_network_golang branch 2 times, most recently from bd4ea50 to bf8c37b Compare September 13, 2018 15:41

amshinde reviewed Sep 14, 2018

View reviewed changes

Sebastien Boeuf added 9 commits September 14, 2018 09:15

netmon: Build netmon from the master Makefile

f131590

This commit modifies the Makefile at the root of this repository so that the binary kata-netmon can be built from there. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

netmon: Add unit testing

55af108

This commit adds some unit testing in order to validate some of the new code that have been introduced with the new network monitor. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

virtcontainers: Start network monitor from virtcontainers

1406d99

This patch enables the code responsible for starting and stopping the network monitor. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

cli: config: Make netmon configurable

0ffe81c

In order to choose if the network monitor should be used or not, this patch makes it configurable from the configuration.toml file. Signed-off-by: Sebastien Boeuf <sebastien.boeuf@intel.com>

sboeuf force-pushed the monitor_network_golang branch from bf8c37b to 0ffe81c Compare September 14, 2018 16:38

amshinde merged commit 40bf149 into kata-containers:master Sep 14, 2018

bergwolf mentioned this pull request Sep 17, 2018

uevent listening goroutine is not (always) working kata-containers/agent#372

Closed


		netlinkFamily = netlink.FAMILY_V4

		storageParentPath = "/var/run/kata-containers/netmon/sbs"

Conversation

sboeuf commented Jul 30, 2018

Uh oh!

sboeuf commented Jul 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sboeuf commented Jul 30, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

katacontainersbot commented Jul 30, 2018

Uh oh!

katacontainersbot commented Jul 30, 2018

Uh oh!

opendev-zuul bot commented Jul 30, 2018

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

katacontainersbot commented Jul 31, 2018

Uh oh!

katacontainersbot commented Jul 31, 2018

Uh oh!

opendev-zuul bot commented Jul 31, 2018

Uh oh!

bergwolf commented Aug 1, 2018

Uh oh!

sboeuf commented Aug 1, 2018

Uh oh!

katacontainersbot commented Aug 1, 2018

Uh oh!

codecov bot commented Aug 1, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

sboeuf commented Jul 30, 2018 •

edited

Loading

codecov bot commented Aug 1, 2018 •

edited

Loading

ghost commented Sep 4, 2018 •

edited by ghost

Loading

bergwolf commented Sep 13, 2018 •

edited by amshinde

Loading