Skip to content

Speed up dockerized builds#28518

Closed
jbeda wants to merge 3 commits intokubernetes:masterfrom
jbeda:remove-copy-output
Closed

Speed up dockerized builds#28518
jbeda wants to merge 3 commits intokubernetes:masterfrom
jbeda:remove-copy-output

Conversation

@jbeda
Copy link
Copy Markdown
Contributor

@jbeda jbeda commented Jul 6, 2016

These changes make the assumption that host volume mounts work. With docker for Mac and docker-machine supporting them there are very few cases left where the build target is truly remote.

With that in mind:

  • We no longer have to copy binaries out.
  • Remove support for boot2docker
  • Stop tar'ing up the sources for the docker image
  • Make sure we cache all of the intermediate build artifacts to speed up incremental builds.

If this passes the sniff test I'll send out a message to kubernetes-dev to let everyone know this is coming and make sure we aren't pulling the rug out from under anyone.


This change is Reviewable

@jbeda jbeda added area/build-release release-note-none Denotes a PR that doesn't merit a release note. labels Jul 6, 2016
@k8s-github-robot k8s-github-robot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 6, 2016
@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Jul 6, 2016

Looks like maybe I broke Jenkins. I'll debug tomorrow.

@jbeda jbeda force-pushed the remove-copy-output branch from ad21fd7 to 33a0f06 Compare July 6, 2016 03:16
@k8s-bot
Copy link
Copy Markdown

k8s-bot commented Jul 6, 2016

Build finished. 3357 tests run, 14 skipped, 0 failed.

@k8s-bot
Copy link
Copy Markdown

k8s-bot commented Jul 6, 2016

Build finished.

@k8s-bot
Copy link
Copy Markdown

k8s-bot commented Jul 6, 2016

Build finished. 330 tests run, 329 skipped, 0 failed.

@luxas
Copy link
Copy Markdown
Member

luxas commented Jul 6, 2016

/subscribe

build/common.sh Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess boot2docker should be removed from here...

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I removed all references to boot2docker.

@luxas
Copy link
Copy Markdown
Member

luxas commented Jul 6, 2016

I think it's worthy a release note since it breaks boot2docker

@k8s-github-robot k8s-github-robot added the kind/design Categorizes issue or PR as related to design. label Jul 6, 2016
@jbeda jbeda force-pushed the remove-copy-output branch from 8d52408 to ab29494 Compare July 6, 2016 22:05
@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Jul 6, 2016

With respect to release note -- this will have zero impact on end users. It won't impact the release at all. I think that we want to announce it widely to everyone working on kubernetes itself but I don't think it needs a release note.

@lavalamp lavalamp assigned zmerlynn and unassigned lavalamp Jul 6, 2016
@lavalamp
Copy link
Copy Markdown
Contributor

lavalamp commented Jul 6, 2016

release notes are for things that affect end users, not developers, so I think release-note-none and a thread on kubernetes-dev are the way to go.

@jessfraz
Copy link
Copy Markdown
Contributor

jessfraz commented Jul 7, 2016

this is neat :)

@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Jul 7, 2016

@thockin -- wanted to make sure you saw this.

@smarterclayton
Copy link
Copy Markdown
Contributor

Moderately impactful for folks who don't run the VMs shipped by Docker, but it's not a deal breaker (it's just more config to make the default paths work, which most people do).

@MHBauer
Copy link
Copy Markdown
Contributor

MHBauer commented Jul 7, 2016

I'm running docker for mac (d4m)

Version 1.12.0-rc2-beta17 (build: 9779)
ff18c0c63c5ff3c4a4a925d191d5592d655779d7

and I do not see a file sharing tab. Do you have a sample screenshot of what that looks like?

Secondarily:
The docs mention needing 3G memory for a build. The default memory on d4m(at least for me on a 16G machine) is 2G, so there should be a mention of the slider or increasing memory access in the d4m use case.

@vishh
Copy link
Copy Markdown
Contributor

vishh commented Jul 7, 2016

Just tested this. Thanks for the cleanup @jbeda!
Next step would be to get make [WHAT=<pkgs>] to automatically run inside a docker container.

@thockin
Copy link
Copy Markdown
Member

thockin commented Jul 7, 2016

I have not reviewed it yet, but I am BEGGGGGGGING to not try to merge this until #25978 lands.

At that point, I agree whole-heartedly with @erictune that all builds, including 'make WHAT=foo' should run in Docker, thereby becoming significantly more hermetic.

build/common.sh Outdated
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can't we just use the local _output/local/go/pkg dir to cache incremental builds, and get rid of the data container entirely? We're using docker as a way to get a hermetic, reproducable build with known-good tools. The source itself is in my local dir. The data container is a leftover from a bygone era.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not really -- subtleties here:

  • The way that host directories are shared into the VM and bind mounted into the container is super slow through VirtualBox. This is one of the big things that comes from Docker for Mac
  • Not all of the intermediate files are under _output. The fact that we need to recompile the standard libraries (and if we don't cache the world gets rebuilt) means we need to map /usr/local/go/pkg/linux_amd64_cgo too.

If we knew that bind mounts would be fast enough, perhaps we could make it work. But they are really slow with VirtualBox.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you know how much sympathy I have for "I want to build from a mac" ? :)

But I am a little confused, I thought the data container only carries /go/src/k8s.io/kubernetes/_output anyway - nothing to do with /usr/local/go (excellent point, though). Given that you assert that bind mounts are slow, we still need to copy the source code into a container, or the build will be v.slow (only for Mac people, who should be used to that by now anyway).

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Part of this change does away with the "tar up source and copy it over" and instead moves to a "bind mount source readonly". This may slow down the virtualbox build. I need to test that. I don't remember if the slow part was read/write or just pure read path. I'll test that out.

This change also makes sure that the /usr/local/go stuff is cached between runs. That is critical for getting builds to be fast and incremental.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh. Yes, well, it would be GREAT if we didn't need the data container. Eventually, maybe?

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2016
jbeda added 2 commits July 7, 2016 13:55
- Remove support for boot2docker.
- Remove support for "copying" binaries back from remote docker engine. I doubt anyone is using this and it slows down other parts of the system.
* No longer tar up sources but instead use a read-only host mount.
* Make sure /usr/local/go/pkg/*_cgo is saved/cached across runs
@jbeda jbeda force-pushed the remove-copy-output branch from ab29494 to d8ab401 Compare July 7, 2016 20:55
@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Jul 7, 2016

Rebased. @thockin -- I'll wait until #25978 lands. Is there a timeline there?

@k8s-github-robot k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 7, 2016
@thockin
Copy link
Copy Markdown
Member

thockin commented Jul 8, 2016

Thanks for blocking. part 1 of 3 is in, part 2 is in queue. Part 3 has had partial review, but I will try to get it into queue tomorrow.

@k8s-github-robot
Copy link
Copy Markdown

@jbeda PR needs rebase

@k8s-github-robot k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jul 9, 2016
@k8s-bot
Copy link
Copy Markdown

k8s-bot commented Aug 4, 2016

GCE e2e build/test passed for commit d8ab401.

@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Aug 8, 2016

@thockin -- are you ready for me to pick this up again and rehabilitate it?

@zmerlynn zmerlynn assigned thockin and ixdy and unassigned zmerlynn Aug 8, 2016
@thockin
Copy link
Copy Markdown
Member

thockin commented Aug 8, 2016

go for it. I have no pending build changes at the moment.

On Mon, Aug 8, 2016 at 9:25 AM, Joe Beda notifications@github.com wrote:

@thockin https://github.com/thockin -- are you ready for me to pick
this up again and rehabilitate it?


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#28518 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AFVgVJlJZ0nH6De0RXxNyezoJLHHwDoLks5qd1h7gaJpZM4JFqEF
.

@ixdy
Copy link
Copy Markdown
Contributor

ixdy commented Aug 8, 2016

cc @spxtr

@luxas
Copy link
Copy Markdown
Member

luxas commented Aug 13, 2016

@jbeda Needs a rebase. Would be great to have faster dockerized builds...

@MHBauer
Copy link
Copy Markdown
Contributor

MHBauer commented Aug 16, 2016

Not much to add. I'll give it a run after the rebase.


Reviewed 5 of 7 files at r1, 7 of 7 files at r2.
Review status: all files reviewed at latest revision, 2 unresolved discussions, some commit checks failed.


build/common.sh, line 248 [r2] (raw file):

      echo "    - Create and start your VM using docker-machine or docker for Mac: "
      echo "      - docker-machine create -d ${DOCKER_MACHINE_DRIVER} ${DOCKER_MACHINE_NAME}"
      echo "      - docker for Mac: check docker icon in menu bar"

maybe docker for Mac: run the Docker app and check for the Docker icon in the menu bar.
If the icon isn't there, the app needs to be run.


cluster/mesos/docker/util.sh, line 81 [r2] (raw file):

  if [[ "${entrypoint}" = "./"* ]]; then
    # relative to project root
    entrypoint="/go/src/github.com/GoogleCloudPlatform/kubernetes/${entrypoint}"

This doesn't seem right. I know you didn't change it, but I saw it out the corner of my eye. Is this function even used?


Comments from Reviewable

@jbeda jbeda mentioned this pull request Aug 17, 2016
8 tasks
@jbeda
Copy link
Copy Markdown
Contributor Author

jbeda commented Aug 19, 2016

Closing this in favor of #30787. @MHBauer -- I'm going to look at your comments in light of that PR.

@jbeda jbeda closed this Aug 19, 2016
k8s-github-robot pushed a commit that referenced this pull request Oct 6, 2016
Automatic merge from submit-queue

Speed up dockerized builds

This PR speeds up dockerized builds.  First, we make sure that we are as incremental as possible.  The bigger change is that now we use rsync to move sources into the container and get data back out.

To do yet:
* [x] Add a random password to rsync.  This is 128bit MD4, but it is better than nothing.
* [x] Lock down rsync to only come from the host.
* [x] Deal with remote docker engines -- this should be necessary for docker-machine on the mac.
* [x] Allow users to specify the port for the rsync daemon.  Perhaps randomize this or let docker pick an ephemeral port and detect the port?
* [x] Copy back generated files so that users can check them in.  This is done for `zz_generated.*` files generated by `make generated_files` 
  * [x] This should include generated proto files so that we can remove the hack-o-rama that is `hack/hack/update-*-dockerized.sh` 
* [x] Start "versioning" the build container and the data container so that the CI system doesn't have to be manually kicked.
* [x] Get some benchmarks to qualify how much faster.

This replaces #28518 and is related to #30600.

cc @thockin @spxtr @david-mcmahon @MHBauer 

Benchmarks by running `make clean ; sync ; time bash -xc 'time build/make-build-image.sh ; time sync ; time build/run.sh make ; time sync; time build/run.sh make'` on a GCE n1-standard-8 with PD-SSD.

| setup | build image | sync | first build | sync | second build | total |
|-------|-------------|----- |----------|------|--------------|------|
| baseline | 0m11.420s | 0m0.812s | 7m2.353s | 0m42.380s | 7m8.381s | 15m5.348s |
| this pr | 0m10.977s | 0m15.168s | 7m31.096s | 1m55.692s | 0m16.514s | 10m9.449s |
@jbeda jbeda deleted the remove-copy-output branch October 13, 2016 18:42
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/build-release kind/design Categorizes issue or PR as related to design. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. release-note-none Denotes a PR that doesn't merit a release note. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.