Use dobi for build tasks #28952

dnephin · 2016-11-29T21:49:59Z

Replaces #27359

Changes:

makefile targets will download the linux version of dobi to run the dobi task
adds Use a separate docker image for cross compiling #26389
removes the Jenkinsfile

Independent images for different (groups of) build tasks

Instead of a single monolithic Dockerfile for every build task, we can split them into separate Dockerfiles for groups of related tasks. This comes with a few benefits:

running many task is significantly faster. For example, running the validate/all task in a container, after a fresh checkout (or after a dependency in the image changes) only takes ~5-10s. Currently it can take easily 10+ minutes to build the monolithic image. Running tests or building the linux binary will also be faster, because we don't need to install all the dependencies required for other tasks (ex Use a separate docker image for cross compiling #26389). This is not just a one-time improvement. Every time we make a dependency change to an image we invalidate everything after it. By splitting the images, a change to one doesn't force us to rebuild the image for all the other tasks. We end up with much better cache hit rates.
maintaining dockerfiles for different platforms becomes easier. We don't need to install dependencies for every task. We only need to install dependencies for building the binary on that platform.

This PR adds two new images (from Dockerfile.validate, and Dockerfile.build), #26389 has one for cross, and there is already a separate image for man page generation. In the near future we can have one for generating the swagger types (#27117). We might also want images for build-rpm and build-dep.

dobi for running build tasks

dobi is a tool I've been working on for "build automation". It is designed based on my experienced writing Makefiles both here and at previous employer, as well as a lot of feedback from Compose users who are attempting to use Compose as a Makefile replacement (which it is not designed to be). dobi is aimed at being a make-like tool that exposes docker resources from a declarative config file. It would eventually be a replacement for the Makefile.

As you can see from the initial dobi.yaml in this PR, you define each resource (mount, image, job) and you can execute them as you would a make target with dobi <resource name>. There are also "actions" for each task, so to remove the build image you could run dobi builder:rm (see the task reference).

dobi comes with a long list of improvements over a Makefile:

significantly better caching (http://dnephin.github.io/dobi/tasks.html#build-default)
more intuitive syntax (I'm probably biased, but it does at least seem to be significantly less verbose than the Makefile version)

If you're interested in trying it out , you can run dobi list to get a list of all the resources that have descriptions.

duglin · 2016-11-30T01:14:01Z

Just wondering... could a similar net effect be done by making independent Dockerfiles for each major step in the build process and then one main Dockerfile that INCLUDEs them?

AkihiroSuda · 2016-11-30T05:14:05Z

Makefile

Can you support other archs and OSes as well?

I agree with @AkihiroSuda, we have to make sure we support other archs and OSes here 👼

What platforms do we support?

Windows, Linux and Mac for sure. Anybody contributing to docker/docker should still be able to build it and such so we shouldn't break them.

And Solaris? (when "runZ" is made available)

Something like that should work (couldn't test, and I am not sure about how chmod +x .. works on windows.. 😅).

diff --git a/Makefile b/Makefile index 035be0a..09eff52 100644 --- a/Makefile +++ b/Makefile @@ -73,6 +73,23 @@ ifeq ($(INTERACTIVE), 1) DOCKER_FLAGS += -t endif +dobicmd := "dobi" +ifeq ($(OS),Windows_NT) + uname_S := Windows +else + uname_S := $(shell uname -s) +endif +ifeq ($(uname_S), Windows) + dobicmd = "dobi.exe" + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-windows.exe" +else ifeq ($(uname_S), Darwin) + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-darwin" +else ifeq ($(uname_S), Linux) + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-linux" +else + $(error unsupported platform) +endif + DOCKER_RUN_DOCKER := $(DOCKER_FLAGS) "$(DOCKER_IMAGE)" default: binary @@ -125,21 +142,21 @@ test-docker-py: build ## run the docker-py tests test-integration-cli: build ## run the integration tests $(DOCKER_RUN_DOCKER) hack/make.sh build-integration-test-binary dynbinary test-integration-cli -dobi: - curl -LsS -o dobi https://github.com/dnephin/dobi/releases/download/v0.8/dobi-linux - chmod +x dobi +$(dobicmd): + curl -LsS -o $(dobicmd) $(dobirelease) + chmod +x $(dobicmd) -test-unit: dobi ## run the unit tests - ./dobi test-unit +test-unit: $(dobicmd) ## run the unit tests + ./$(dobicmd) test-unit tgz: build cross ## build the archives (.zip on windows and .tgz\notherwise) containing the binaries $(DOCKER_RUN_DOCKER) hack/make.sh dynbinary binary tgz -validate: dobi ## run full validation - ./dobi validate +validate: $(dobicmd) ## run full validation + ./$(dobicmd) validate -cross: bundles dobi ## cross build the binaries for darwin, freebsd and\nwindows - ./dobi cross +cross: bundles $(dobicmd) ## cross build the binaries for darwin, freebsd and\nwindows + ./$(dobicmd) cross win: build ## cross build the binary for windows $(DOCKER_RUN_DOCKER) hack/make.sh win

Definitely +1 on finding a way to make this more forgiving of other platforms. 😄

Any invocation of curl probably ought to include -f so that it passes failure forward to Make (and the build doesn't barrel onwards without dobi actually downloaded).

AkihiroSuda · 2016-11-30T05:14:23Z

Makefile

binary is gone?

Oops, this was a bad rebase, thanks

AkihiroSuda · 2016-11-30T05:17:17Z

Makefile

init-go-pkg-cache is not handled in dobi?

I restored this, it was a bad rebase as well. It should still work just fine.

glensc · 2016-11-30T08:06:08Z

Makefile

cleanup \n in comments (replace with space or real newline).

i reported this in your previous PR: #26389 (comment)

These \n are correct. It's used by make help.

vdemeester · 2016-11-30T08:13:30Z

@duglin yes without some benefits of dobi and with probably adding a bit complexity in our Makefile (but I am not sure 👼)

vdemeester · 2016-11-30T16:11:01Z

@dnephin 23:23:47 error: binary and cross must be run before tgz on janky (and I guess experimental)

dnephin · 2016-11-30T17:48:07Z

could a similar net effect be done by making independent Dockerfiles for each major step in the build process and then one main Dockerfile that INCLUDEs them?

It would depend on the implementation of INCLUDE, but I don't think so. For INCLUDE to work the way it does in dobi you'd have to support a DAG for caching layers, instead of a single linear sequence. I don't think that is possible when the output is an image with a single root filesystem. Some includes will add files to directories used by other includes. They may even touch the same files.

The Makefile can do some of what dobi does, in a much less readable way. That's not surprising because dobi was designed to replace it, so the initial feature set was modelled after this kind of Makefile.

Signed-off-by: Daniel Nephin <dnephin@docker.com>

dnephin · 2016-11-30T17:50:18Z

@vdemeester yes, since I pulled in #26389 this requires Jenkins config changes again. This is one reason I combined this way the Jenkinsfile change initially. To make this work correctly for CI we really need Jenkinsfile.

vdemeester · 2016-12-01T13:15:09Z

Hum… So the process or using Jenkinsfile will take some time to make it right… I think we should have a build that is driven by a Jenkinsfile without breaking the current build at first (so we can easily go back if needed).

The quicker we have a least a build driven by Jenkinsfile the quicker we can validate that it will fit and get feedback and experiment with it. But on the other hand we really shouldn't break the current flow unless we are sure we won't have to go back. So I would think we should define new "tasks" for now (experimental tasks you might say) and overtime, migrate to them.

icecrime · 2016-12-07T22:28:06Z

I'm +1 on design. Ping @tianon, WDYT?

cpuguy83 · 2017-01-14T21:53:17Z

ping @tianon

vdemeester

Overall looks good but I wonder how this will fit for the windows build (that uses powershell and another makefile cc @johnstep @jhowardmsft).

Thus my diff on the Makefile might be a little bit overcomplicated but.. A side-note is that now, we would have some "unsupported" platform where it's not possible to build as-is docker/docker. Right now, as long as you have docker installed, it should work.

Another thing, not sure if it's a dobi bug or else, but if I run make test-unit, I can do any CTRL-C I want, it won't cancel the current job.

/cc @icecrime @tonistiigi @tiborvass on how this would fit in some possible builder evolution 👼

vdemeester · 2017-01-18T10:51:38Z

Makefile

Something like that should work (couldn't test, and I am not sure about how chmod +x .. works on windows.. 😅).

diff --git a/Makefile b/Makefile index 035be0a..09eff52 100644 --- a/Makefile +++ b/Makefile @@ -73,6 +73,23 @@ ifeq ($(INTERACTIVE), 1) DOCKER_FLAGS += -t endif +dobicmd := "dobi" +ifeq ($(OS),Windows_NT) + uname_S := Windows +else + uname_S := $(shell uname -s) +endif +ifeq ($(uname_S), Windows) + dobicmd = "dobi.exe" + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-windows.exe" +else ifeq ($(uname_S), Darwin) + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-darwin" +else ifeq ($(uname_S), Linux) + dobirelease = "https://github.com/dnephin/dobi/releases/download/v0.8/dobi-linux" +else + $(error unsupported platform) +endif + DOCKER_RUN_DOCKER := $(DOCKER_FLAGS) "$(DOCKER_IMAGE)" default: binary @@ -125,21 +142,21 @@ test-docker-py: build ## run the docker-py tests test-integration-cli: build ## run the integration tests $(DOCKER_RUN_DOCKER) hack/make.sh build-integration-test-binary dynbinary test-integration-cli -dobi: - curl -LsS -o dobi https://github.com/dnephin/dobi/releases/download/v0.8/dobi-linux - chmod +x dobi +$(dobicmd): + curl -LsS -o $(dobicmd) $(dobirelease) + chmod +x $(dobicmd) -test-unit: dobi ## run the unit tests - ./dobi test-unit +test-unit: $(dobicmd) ## run the unit tests + ./$(dobicmd) test-unit tgz: build cross ## build the archives (.zip on windows and .tgz\notherwise) containing the binaries $(DOCKER_RUN_DOCKER) hack/make.sh dynbinary binary tgz -validate: dobi ## run full validation - ./dobi validate +validate: $(dobicmd) ## run full validation + ./$(dobicmd) validate -cross: bundles dobi ## cross build the binaries for darwin, freebsd and\nwindows - ./dobi cross +cross: bundles $(dobicmd) ## cross build the binaries for darwin, freebsd and\nwindows + ./$(dobicmd) cross win: build ## cross build the binary for windows $(DOCKER_RUN_DOCKER) hack/make.sh win

dnephin · 2017-01-18T16:10:01Z

I wonder how this will fit for the windows build (that uses powershell and another makefile)

The dobi.yml should really be replacing the makefiles entirely, so we would just have some dobi targets to build the windows binary (binary-win, build-win, etc). The powershell should really be running inside a container, not on the host, in the same way that our bash scripts should run in a linux container.

we would have some "unsupported" platform where it's not possible to build as-is docker/docker. Right now, as long as you have docker installed, it should work.

Why would we have unsupported platforms? dobi can be built for a lot more platforms than docker/docker. Right now you need make and docker installed. This PR just starts the transition to require make with dobi. The number of dependencies doesn't really change.

Another thing, not sure if it's a dobi bug or else, but if I run make test-unit, I can do any CTRL-C I want, it won't cancel the current job.

That sounds like the wrong behaviour to me. If I ctrl-c something I expect it to stop. Either way, both options are possible with dobi. If you use interactive: true the ctrl-c will work, if it's non-interactive it doesn't.

vdemeester · 2017-01-18T16:16:21Z

That sounds like the wrong behaviour to me. If I ctrl-c something I expect it to stop. Either way, both options are possible with dobi. If you use interactive: true the ctrl-c will work, if it's non-interactive it doesn't.

Me too. It is stopping on master, it is not stopping on this PR 👼.

lowenna · 2017-01-18T17:46:04Z

If this PR is entirely for cross-building, and packaging for the "official release builds" and doesn't affect the native Windows building mechanisms, then I'll leave this to the rest of you guys to figure out 😇 I'm not sure there was a direct question for me above. That said:

powershell should really be running inside a container, not on the host, in the same way that our bash scripts should run in a linux container.

Urrrrrm - the PS does run inside a container under CI. And also can be run outside on the host for development purposes. Snipped from https://github.com/docker/docker/blob/master/Dockerfile.windows#L65-L75:

...
# 3. Build a docker image with the components required to build the docker binaries from source
#    by running one of the following:
#
#    >>   docker build -t nativebuildimage -f Dockerfile.windows .          
#
#
# 4. Build the docker executable binaries by running one of the following:
#
#    >>   docker run --name binaries nativebuildimage hack\make.ps1 -Binary
...

Indeed if you build natively on Windows outside of a container, you get this with the warning similar to Linux), which you won't see in any of the Jenkins RS1 runs or when building in a container.

E:\go\src\github.com\docker\docker [master]> hack\make.ps1 -binary
INFO: make.ps1 starting at 01/18/2017 09:38:31

WARNING: Not running in a container. The result might be an incorrect build.

INFO: Invoking autogen...
INFO: Building daemon...
INFO: Building client...

 ________   ____  __.
 \_____  \ |    |/ _|
 /   |   \|      <
 /    |    \    |  \
 \_______  /____|__ \
         \/        \/

INFO: make.ps1 ended at 01/18/2017 09:39:24
E:\go\src\github.com\docker\docker [master]>

I should also point out that I've yet to find a native (non-Cygwin/MSYS2-based) standalone make.exe that works on nanoserver based hosts, hence one reason why a makefile is not used on Windows natively.

dnephin · 2017-01-18T18:03:19Z

the PS does run inside a container under CI. And also can be run outside on the host for development purposes.

Cool, I didn't mean to suggest that it wasn't, just that dobi could make it easier to build the windows version from a linux host. If a developer has a windows VM setup with docker runner in the VM, dobi could be run from the linux host to build the windows binary (as long as DOCKER_HOST was set to point at the docker engine in the VM). This is basically the inverse of what machine/toolbox are doing right now for linux docker on windows.

Personally I would find that a lot nicer to work with. I wouldn't have to worry about setting up a windows development environment or dealing with PS. I could run dobi build-win from my host, and get a windows binary in bundles/ somewhere. make shouldn't be necessary at all in the scenario.

I'm not sure if that's possible today or not. At the very least it doesn't seem to be possible with a single command.

lowenna · 2017-01-18T18:14:15Z

Gotcha. From a Linux developer perspective hacking on docker wanting to build a Windows binary, or make sure they haven't broken the Windows build, that makes entire sense as that's their native environment.

From a Windows developer wanting to do native development, the same way a Linux developer generally doesn't have a Windows VM running 24x7, a Windows developer will probably not have a Linux VM running 24x7, so needs to continue to be able to do native builds in some way.

tianon

Generally design seems OK to me 👍

tianon · 2017-01-18T18:34:19Z

Makefile

Definitely +1 on finding a way to make this more forgiving of other platforms. 😄

Any invocation of curl probably ought to include -f so that it passes failure forward to Make (and the build doesn't barrel onwards without dobi actually downloaded).

tianon · 2017-01-18T18:39:36Z

hack/make/.platforms

+    linux/386 linux/arm
+	darwin/amd64
+	freebsd/amd64 freebsd/386 freebsd/arm
+	windows/amd64 windows/386"


Indentation here is a little odd (mixed tabs/spaces) -- also, I think this might need to be conditionally set based on whether it's set from the environment (docker run -e DOCKER_CROSSPLATFORMS=... docker-dev... is supported currently), ie:

: ${DOCKER_CROSSPLATFORMS:=" linux/386 linux/arm darwin/amd64 freebsd/amd64 freebsd/386 freebsd/arm windows/amd64 windows/386 "} export DOCKER_CROSSPLATFORMS

AkihiroSuda · 2017-01-19T01:50:34Z

Is is acceptable to make make depends on go, and use go get for installing dobi, without a mess of uname?
For pinning the version, gopkg.in can be used.

LK4D4 · 2017-01-30T20:41:16Z

Ok, this has problems with platforms support. Am I right, @dnephin ?

dnephin · 2017-01-30T21:09:04Z

Yes, I think that's right.

If we make dobi the entrypoint it's pretty easy. I can just publish versions of dobi for all supported platforms (using gox) and that's it.

If we want to keep the Makefile as the entrypoint there needs to be a script which downloads the correct version for each platform.

Personally I don't think it makes much sense to keep the Makefile.

LK4D4 · 2017-02-01T17:37:23Z

@dnephin I don't care about makefile too :) Not sure about all other people.

vdemeester · 2017-03-01T14:13:56Z

ok, let's close this for now 👼 🙏

GordonTheTurtle added the status/0-triage label Nov 29, 2016

This was referenced Nov 29, 2016

Use a separate docker image for cross compiling #26389

Closed

PoC: Jenkinsfile+dobi for CI #27359

Closed

vdemeester self-assigned this Nov 29, 2016

vdemeester added status/1-design-review and removed status/0-triage labels Nov 29, 2016

AkihiroSuda reviewed Nov 30, 2016

View reviewed changes

glensc reviewed Nov 30, 2016

View reviewed changes

dnephin added 2 commits November 30, 2016 12:49

Add a dobi.yaml for a few project tasks

645c016

Signed-off-by: Daniel Nephin <dnephin@docker.com>

Dockerfile for cross compiling.

9965199

Signed-off-by: Daniel Nephin <dnephin@docker.com>

dnephin force-pushed the use-dobi branch from 3151e14 to 9965199 Compare November 30, 2016 17:50

AkihiroSuda mentioned this pull request Dec 5, 2016

[WIP, RFC] new make target: test-integration-cli-parallel #26684

Closed

vdemeester suggested changes Jan 18, 2017

View reviewed changes

tianon reviewed Jan 18, 2017

View reviewed changes

vdemeester closed this Mar 1, 2017

Use dobi for build tasks #28952

Use dobi for build tasks #28952

Uh oh!

Conversation

dnephin commented Nov 29, 2016

Independent images for different (groups of) build tasks

dobi for running build tasks

Uh oh!

duglin commented Nov 30, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vdemeester commented Nov 30, 2016

Uh oh!

vdemeester commented Nov 30, 2016

Uh oh!

dnephin commented Nov 30, 2016

Uh oh!

dnephin commented Nov 30, 2016

Uh oh!

vdemeester commented Dec 1, 2016

Uh oh!

icecrime commented Dec 7, 2016

Uh oh!

cpuguy83 commented Jan 14, 2017

Uh oh!

vdemeester left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dnephin commented Jan 18, 2017

Uh oh!

vdemeester commented Jan 18, 2017

Uh oh!

lowenna commented Jan 18, 2017

Uh oh!

dnephin commented Jan 18, 2017

Uh oh!

lowenna commented Jan 18, 2017

Uh oh!

tianon left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

AkihiroSuda commented Jan 19, 2017

Uh oh!

LK4D4 commented Jan 30, 2017

Uh oh!

dnephin commented Jan 30, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

vdemeester left a comment •

edited

Loading

dnephin commented Jan 30, 2017 •

edited

Loading