Speed up dockerized builds by jbeda · Pull Request #30787 · kubernetes/kubernetes

jbeda · 2016-08-17T17:56:39Z

This PR speeds up dockerized builds. First, we make sure that we are as incremental as possible. The bigger change is that now we use rsync to move sources into the container and get data back out.

To do yet:

Add a random password to rsync. This is 128bit MD4, but it is better than nothing.
Lock down rsync to only come from the host.
Deal with remote docker engines -- this should be necessary for docker-machine on the mac.
Allow users to specify the port for the rsync daemon. Perhaps randomize this or let docker pick an ephemeral port and detect the port?
Copy back generated files so that users can check them in. This is done for zz_generated.* files generated by make generated_files
- This should include generated proto files so that we can remove the hack-o-rama that is hack/hack/update-*-dockerized.sh
Start "versioning" the build container and the data container so that the CI system doesn't have to be manually kicked.
Get some benchmarks to qualify how much faster.

This replaces #28518 and is related to #30600.

cc @thockin @spxtr @david-mcmahon @MHBauer

Benchmarks by running make clean ; sync ; time bash -xc 'time build/make-build-image.sh ; time sync ; time build/run.sh make ; time sync; time build/run.sh make' on a GCE n1-standard-8 with PD-SSD.

setup	build image	sync	first build	sync	second build	total
baseline	0m11.420s	0m0.812s	7m2.353s	0m42.380s	7m8.381s	15m5.348s
this pr	0m10.977s	0m15.168s	7m31.096s	1m55.692s	0m16.514s	10m9.449s

This change is

spxtr · 2016-08-19T06:14:18Z

Roughly how much faster is it?

jbeda · 2016-08-19T15:57:00Z

I'll pull together some benchmarks before I remove the WIP label.

thockin · 2016-08-19T23:39:39Z

build/common.sh

-      chown -R $(id -u).$(id -g) "${REMOTE_OUTPUT_ROOT}"
+      chown -R $(id -u).$(id -g)
+        "${REMOTE_ROOT}"
+        /usr/local/go/pkg/linux_386_cgo


why not just do all of /usr/local/go/pkg/

That ends up whiting out the compiler:

$ make WHAT='cmd/kubectl' +++ [0820 16:28:15] Generating bindata: /go/src/k8s.io/kubernetes/test/e2e/framework/gobindata_util.go ../../../hack/../test/e2e/generated/bindata.go.tmp ../../../hack/../test/e2e/generated/bindata.go differ: char 1523, line 29 +++ [0820 16:28:16] Building the toolchain targets: k8s.io/kubernetes/hack/cmd/teststale go tool: no such tool "compile" Makefile.generated_files:279: recipe for target '_output/bin/deepcopy-gen' failed make[1]: *** [_output/bin/deepcopy-gen] Error 1 Makefile:281: recipe for target 'generated_files' failed make: *** [generated_files] Error 2

to just chown? Or is that from volume mounting on pkg, which I could understand.

The compiler itself is under /usr/local/go/pkg. If we make that a volume it gets whited out. Not sure what else to do there.

We could do something where we move it aside when building the image and then copy it back as part of the chown step, but that seems complicated too.

I meant just to chmod/chown the whole thing, not to volume mount the whole
thing.

On Mon, Aug 22, 2016 at 2:24 PM, Joe Beda notifications@github.com wrote:

In build/common.sh
#30787 (comment)
:

--volume "${REMOTE_OUTPUT_GOPATH}" # make a non-root owned mountpoint "${KUBE_BUILD_IMAGE}"

chown -R $(id -u).$(id -g) "${REMOTE_OUTPUT_ROOT}"

chown -R $(id -u).$(id -g)

"${REMOTE_ROOT}"

/usr/local/go/pkg/linux_386_cgo

The compiler itself is under /usr/local/go/pkg. If we make that a volume
it gets whited out. Not sure what else to do there.

We could do something where we move it aside when building the image and
then copy it back as part of the chown step, but that seems complicated too.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/pull/30787/files/c0ba418d1f5f07f74ae8e9aa392e7ddbcfa0a144#r75761703,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVEMQP4zlhDmbqAsqoZD3z7Rl7Y3Yks5qihN5gaJpZM4Jmt7e
.

That seems to work. Latest versions have that.

thockin · 2016-08-22T06:22:23Z

cmd/libs/go2idl/conversion-gen/generators/conversion.go

 		if !pkgNeedsGeneration {
-			glog.V(5).Infof("  no viable conversions, not generating for this package")
-			continue
+			glog.Errorf("Package %v has 'k8s:conversion-gen' tags but no viable conversions", i)


seems unrelated...

The fact that this wasn't outputting a file was confusing your generated files make system. I wasn't able to get to a "no op" make without some sort of fix here.

?? How do I repro that at head?

On Mon, Aug 22, 2016 at 6:18 AM, Joe Beda notifications@github.com wrote:

In cmd/libs/go2idl/conversion-gen/generators/conversion.go
#30787 (comment)
:

@@ -280,8 +280,7 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
}
}
if !pkgNeedsGeneration {

glog.V(5).Infof(" no viable conversions, not generating for this package")

continue

glog.Errorf("Package %v has 'k8s:conversion-gen' tags but no viable conversions", i)

The fact that this wasn't outputting a file was confusing your generated
files make system. I wasn't able to get to a "no op" make without some sort
of fix here.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/pull/30787/files/9269652b4fc9760754e0b4e47cc2cb995f9d3e24#r75674739,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVM2XusCz26zLyZDn5qPbaCQ91KPbks5qiaGdgaJpZM4Jmt7e
.

It's pretty fast but still extra work.

repo

[jbeda@k8s-build:~/src/kubernetes] (master)$ make clean +++ [0822 18:08:36] Verifying Prerequisites.... !!! [0822 18:08:36] Build image not built. Cannot clean via docker build image. +++ [0822 18:08:36] Removing data container kube-build-data-27932f596f +++ [0822 18:08:36] Removing _output directory +++ [0822 18:08:36] Deleting docker image kube-build:build-27932f596f +++ [0822 18:08:36] Cleaning all other untagged docker images :) Mon Aug 22 18:08:36 [jbeda@k8s-build:~/src/kubernetes] (master)$ make generated_files +++ [0822 18:08:49] Generating bindata: /home/jbeda/src/kubernetes/test/e2e/framework/gobindata_util.go +++ [0822 18:08:49] Building the toolchain targets: k8s.io/kubernetes/hack/cmd/teststale +++ [0822 18:08:50] Building go targets for linux/amd64: cmd/libs/go2idl/deepcopy-gen +++ [0822 18:08:55] Generating bindata: /home/jbeda/src/kubernetes/test/e2e/framework/gobindata_util.go +++ [0822 18:08:56] Building the toolchain targets: k8s.io/kubernetes/hack/cmd/teststale +++ [0822 18:08:56] Building go targets for linux/amd64: cmd/libs/go2idl/conversion-gen E0822 18:09:00.238473 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/apps/v1alpha1.PetSetSpec <-> k8s.io/kubernetes/pkg/apis/apps.PetSetSpec E0822 18:09:00.258650 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.HorizontalPodAutoscalerSpec <-> k8s.io/kubernetes/pkg/apis/autoscaling.HorizontalPodAutoscalerSpec E0822 18:09:00.266875 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.JobSpec <-> k8s.io/kubernetes/pkg/apis/batch.JobSpec E0822 18:09:00.282497 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.RollingUpdateDeployment <-> k8s.io/kubernetes/pkg/apis/extensions.RollingUpdateDeployment E0822 18:09:00.285592 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.ScaleStatus <-> k8s.io/kubernetes/pkg/apis/extensions.ScaleStatus :) Mon Aug 22 18:09:00 [jbeda@k8s-build:~/src/kubernetes] (master)$ make generated_files --debug=b GNU Make 4.0 Built for x86_64-pc-linux-gnu Copyright (C) 1988-2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Reading makefiles... Updating goal targets.... File 'generated_files' does not exist. Must remake target 'generated_files'. GNU Make 4.0 Built for x86_64-pc-linux-gnu Copyright (C) 1988-2013 Free Software Foundation, Inc. License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html> This is free software: you are free to change and redistribute it. There is NO WARRANTY, to the extent permitted by law. Reading makefiles... Updating goal targets.... File 'generated_files' does not exist. File 'gen_deepcopy' does not exist. Must remake target 'gen_deepcopy'. Successfully remade target file 'gen_deepcopy'. File 'gen_conversion' does not exist. File 'federation/apis/core/v1/zz_generated.conversion.go' does not exist. Must remake target 'federation/apis/core/v1/zz_generated.conversion.go'. Successfully remade target file 'federation/apis/core/v1/zz_generated.conversion.go'. Must remake target 'gen_conversion'. Successfully remade target file 'gen_conversion'. Must remake target 'generated_files'. Successfully remade target file 'generated_files'. Successfully remade target file 'generated_files'.

ACK. Good catch.

On Mon, Aug 22, 2016 at 11:13 AM, Joe Beda notifications@github.com wrote:

In cmd/libs/go2idl/conversion-gen/generators/conversion.go
#30787 (comment)
:

@@ -280,8 +280,7 @@ func Packages(context *generator.Context, arguments *args.GeneratorArgs) generat
}
}
if !pkgNeedsGeneration {

glog.V(5).Infof(" no viable conversions, not generating for this package")

continue

glog.Errorf("Package %v has 'k8s:conversion-gen' tags but no viable conversions", i)

It's pretty fast but still extra work.

repo

jbeda@k8s-build:~/src/kubernetes$ make clean
+++ [0822 18:08:36] Verifying Prerequisites....
!!! [0822 18:08:36] Build image not built. Cannot clean via docker build image.
+++ [0822 18:08:36] Removing data container kube-build-data-27932f596f
+++ [0822 18:08:36] Removing _output directory
+++ [0822 18:08:36] Deleting docker image kube-build:build-27932f596f
+++ [0822 18:08:36] Cleaning all other untagged docker images
:) Mon Aug 22 18:08:36
jbeda@k8s-build:~/src/kubernetes$ make generated_files
+++ [0822 18:08:49] Generating bindata:
/home/jbeda/src/kubernetes/test/e2e/framework/gobindata_util.go
+++ [0822 18:08:49] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
+++ [0822 18:08:50] Building go targets for linux/amd64:
cmd/libs/go2idl/deepcopy-gen
+++ [0822 18:08:55] Generating bindata:
/home/jbeda/src/kubernetes/test/e2e/framework/gobindata_util.go
+++ [0822 18:08:56] Building the toolchain targets:
k8s.io/kubernetes/hack/cmd/teststale
+++ [0822 18:08:56] Building go targets for linux/amd64:
cmd/libs/go2idl/conversion-gen
E0822 18:09:00.238473 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/apps/v1alpha1.PetSetSpec <-> k8s.io/kubernetes/pkg/apis/apps.PetSetSpec
E0822 http://k8s.io/kubernetes/pkg/apis/apps.PetSetSpecE0822 18:09:00.258650 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.HorizontalPodAutoscalerSpec <-> k8s.io/kubernetes/pkg/apis/autoscaling.HorizontalPodAutoscalerSpec
E0822 http://k8s.io/kubernetes/pkg/apis/autoscaling.HorizontalPodAutoscalerSpecE0822 18:09:00.266875 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.JobSpec <-> k8s.io/kubernetes/pkg/apis/batch.JobSpec
E0822 http://k8s.io/kubernetes/pkg/apis/batch.JobSpecE0822 18:09:00.282497 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.RollingUpdateDeployment <-> k8s.io/kubernetes/pkg/apis/extensions.RollingUpdateDeployment
E0822 http://k8s.io/kubernetes/pkg/apis/extensions.RollingUpdateDeploymentE0822 18:09:00.285592 8625 conversion.go:589] Warning: could not generate autoConvert functions for k8s.io/kubernetes/pkg/apis/extensions/v1beta1.ScaleStatus <-> k8s.io/kubernetes/pkg/apis/extensions.ScaleStatus
:) Mon Aug 22 18:09:00
jbeda@k8s-build:~/src/kubernetes$ make generated_files --debug=b
GNU Make 4.0
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Updating goal targets....
File 'generated_files' does not exist.
Must remake target 'generated_files'.
GNU Make 4.0
Built for x86_64-pc-linux-gnu
Copyright (C) 1988-2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Reading makefiles...
Updating goal targets....
File 'generated_files' does not exist.
File 'gen_deepcopy' does not exist.
Must remake target 'gen_deepcopy'.
Successfully remade target file 'gen_deepcopy'.
File 'gen_conversion' does not exist.
File 'federation/apis/core/v1/zz_generated.conversion.go' does not exist.
Must remake target 'federation/apis/core/v1/zz_generated.conversion.go'.
Successfully remade target file 'federation/apis/core/v1/zz_generated.conversion.go'.
Must remake target 'gen_conversion'.
Successfully remade target file 'gen_conversion'.
Must remake target 'generated_files'.
Successfully remade target file 'generated_files'.
Successfully remade target file 'generated_files'.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/kubernetes/kubernetes/pull/30787/files/9269652b4fc9760754e0b4e47cc2cb995f9d3e24#r75728857,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVNEJWxeUo7Q5hXJa-bOF4I1RVespks5qiebagaJpZM4Jmt7e
.

jbeda · 2016-08-23T17:45:34Z

To anyone watching from home, I'm fighting with Jenkins. I was using a Docker 1.12 feature and had to discover that Jenkins was using 1.9.1 and adjust. Now the verification test is failing (can't talk to the rsync container) and I'm trying to figure out why. The only way I have to test is to push a new change with debug commands. It is essentially printf debugging with a 15m+ delay.

I'll clean up the patch set after things look to be working reliably.

MHBauer · 2016-08-25T19:56:13Z

build/README.md

-3. **Optional** [Google Cloud SDK](https://developers.google.com/cloud/sdk/)
+1. Docker, using one of the following configurations:
+  1. **Mac OS X** You can either use Docker for Mac or docker-machine. See installation instructions [here](https://docs.docker.com/installation/mac/).
+     **Note**: You will want to set the Docker VM to have at least 3GB of initial memory or building will likely fail. (See: [#11852]( http://issue.k8s.io/11852)). https://github.com/kubernetes/kubernetes/issues/14773))


Too many closing parens on this line.

make-generated-{protobuf,runtime}.sh was doing some really nasty stuff with how the build container was managed in order to copy results out. Since we have more flexibility to grab results out of the build container, we can now avoid all of this. Ideally we wouldn't have `hack` calling `build` at all, but we aren't there yet.

We weren't getting incremental builds because of new test only subpackages. Our voodoo combo of 'go install' and 'go test -c' didn't cache things like 'test/e2e_node/services'. Add the '-i' flag to 'go test' to install test only dependencies too.

jbeda · 2016-10-04T02:43:31Z

Okay! Please take another look. Hopefully we are there.

ivan4th · 2016-10-04T09:56:59Z

I have slightly unexpected possible use for remote docker mode: don't rsync back binaries after build, use them to start DIND cluster on remote docker instead. E.g. I'm on a not very fast 3G connection and want to use GCE for dev. How hard it's to do this? (at least opt out of rsyncing binaries back)

jbeda · 2016-10-04T19:12:56Z

@ivan4th -- interesting case! Right now we always "copy back" to the host machine but we can explore making that optional after this gets in.

ivan4th · 2016-10-05T09:20:11Z

@jbeda actually it looks like KUBE_RUN_COPY_OUTPUT=n build/run.sh may do the trick.

ixdy

woo! I'm fine with getting this in and then iterating with any additional fixes if necessary. two minor comments, neither of which is blocking.

ixdy · 2016-10-05T23:57:53Z

build/common.sh

  LOCAL_OUTPUT_BUILD_CONTEXT="${LOCAL_OUTPUT_IMAGE_STAGING}/${KUBE_BUILD_IMAGE}"
+
+  kube::version::get_version_vars
+  kube::version::save_version_vars "${KUBE_ROOT}/.dockerized-kube-version-defs"


minor nonblocking suggestion: maybe clean deletes this?

ixdy · 2016-10-06T00:13:39Z

build/lib/release.sh

+# limitations under the License.
+
+# This file creates release artifacts (tar files, container images) that are
+# ready to distribute to install or distribute to end users.


nit: "ready to distribute to install or distribute" seems awkward?

k8s-github-robot · 2016-10-06T02:12:24Z

@k8s-bot test this [submit-queue is verifying that this PR is safe to merge]

k8s-github-robot · 2016-10-06T02:51:09Z

Automatic merge from submit-queue

duglin · 2016-10-06T12:34:18Z

woo hoo!

This was broken by kubernetes#30787. A stray bash `source` caused an undefined variable reference. Apparently the federation images have a parallel nad different "release" path that isn't tested by the pre-checkin tests.

Automatic merge from submit-queue Fix broken build/push-federation-images.sh The federation CI build is broken by #30787. A stray bash `source` caused an undefined variable reference. Apparently the federation images have a parallel nad different "release" path that isn't tested by the pre-checkin tests.

This was broken by kubernetes#30787. A stray bash `source` caused an undefined variable reference. Apparently the federation images have a parallel nad different "release" path that isn't tested by the pre-checkin tests.

@saad-ali

Automatic merge from submit-queue Make kubernetes-src.tar.gz contain source code again **What this PR does / why we need it**: This PR fixes `kubernetes-src.tar.gz` by reviving the logic for `kube-source.tar.gz` from before #30787. **Which issue this PR fixes**: fixes #36403 @saad-ali @jbeda @zmerlynn

googlebot added the cla: yes label Aug 17, 2016

k8s-github-robot assigned ixdy Aug 17, 2016

k8s-github-robot added kind/design Categorizes issue or PR as related to design. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. release-note-label-needed labels Aug 17, 2016

jbeda mentioned this pull request Aug 19, 2016

Speed up dockerized builds #28518

Closed

thockin reviewed Aug 19, 2016
View reviewed changes

jbeda force-pushed the rsync branch from c0ba418 to 2b97e45 Compare August 20, 2016 16:13

k8s-github-robot added size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. and removed size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Aug 20, 2016

thockin reviewed Aug 22, 2016
View reviewed changes

jbeda force-pushed the rsync branch from 4c6efd1 to 2c5040d Compare August 22, 2016 16:29

k8s-github-robot mentioned this pull request Aug 22, 2016

[k8s.io] Kubectl client [k8s.io] Kubectl expose should create services for rc [Conformance] {Kubernetes e2e suite} #29227

Closed

jbeda force-pushed the rsync branch 2 times, most recently from 477bdf5 to 0d4efbc Compare August 22, 2016 21:29

k8s-github-robot mentioned this pull request Aug 23, 2016

[k8s.io] Pods should contain environment variables for services [Conformance] {E2eNode Suite} #30548

Closed

jbeda force-pushed the rsync branch from d65582a to 4060838 Compare August 23, 2016 13:44

This was referenced Aug 23, 2016

[k8s.io] ScheduledJob should replace jobs when ReplaceConcurrent {Kubernetes e2e suite} #30542

Closed

[k8s.io] Probing container should *not* be restarted with a /healthz http liveness probe [Conformance] {E2eNode Suite} #30714

Closed

jbeda force-pushed the rsync branch 4 times, most recently from 6ac6d27 to dd2cac8 Compare August 23, 2016 17:34

jbeda force-pushed the rsync branch from dd2cac8 to b0a9292 Compare August 23, 2016 23:28

jbeda mentioned this pull request Aug 24, 2016

Build system improvements #30600

Closed

jbeda force-pushed the rsync branch from b0a9292 to 5057c0f Compare August 25, 2016 00:12

MHBauer reviewed Aug 25, 2016
View reviewed changes

jbeda added 5 commits October 3, 2016 19:42

Move release code (bash) out of build/common.sh

f861163

Install test subpackages too

dd3ac00

We weren't getting incremental builds because of new test only subpackages. Our voodoo combo of 'go install' and 'go test -c' didn't cache things like 'test/e2e_node/services'. Add the '-i' flag to 'go test' to install test only dependencies too.

Update comment on release.sh to reflect new release repo

034fc88

Remove unused bash variables

d955f54

jbeda force-pushed the rsync branch from 8b7df79 to d955f54 Compare October 4, 2016 02:42

ixdy approved these changes Oct 6, 2016

View reviewed changes

ixdy added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Oct 6, 2016

k8s-github-robot merged commit 63139f9 into kubernetes:master Oct 6, 2016

k8s-github-robot mentioned this pull request Oct 6, 2016

[k8s.io] Downward API volume should update annotations on modification [Conformance] {Kubernetes e2e suite} #34014

Closed

jbeda mentioned this pull request Oct 6, 2016

Fix broken build/push-federation-images.sh #34229

Merged

ixdy mentioned this pull request Oct 6, 2016

e2e flake: "Syncing sources to container" failed in PR builder #34214

Closed

ahakanbaba mentioned this pull request Oct 10, 2016

Permission denied error on .make/all_go_dirs.mk (or .make) during MaxOs Build with docker-machine #33800

Closed

jbeda deleted the rsync branch October 13, 2016 18:42

This was referenced Nov 8, 2016

kubernetes-src.tar.gz contains no source code #36403

Closed

Make kubernetes-src.tar.gz contain source code again #36407

Merged

ixdy mentioned this pull request Feb 24, 2017

New command for stand-alone GKE certificates controller #41160

Merged

Conversation

jbeda commented Aug 17, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

spxtr commented Aug 19, 2016

Uh oh!

jbeda commented Aug 19, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbeda commented Aug 23, 2016 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jbeda commented Oct 4, 2016

Uh oh!

ivan4th commented Oct 4, 2016

Uh oh!

jbeda commented Oct 4, 2016

Uh oh!

ivan4th commented Oct 5, 2016

Uh oh!

ixdy left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

k8s-github-robot commented Oct 6, 2016

Uh oh!

k8s-github-robot commented Oct 6, 2016

Uh oh!

duglin commented Oct 6, 2016

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

jbeda commented Aug 17, 2016 •

edited

Loading

jbeda commented Aug 23, 2016 •

edited

Loading