Skip to content

Simplify build upload, improve testing of builds#900

Merged
creack merged 14 commits into
masterfrom
simpler-build-upload
Jun 21, 2013
Merged

Simplify build upload, improve testing of builds#900
creack merged 14 commits into
masterfrom
simpler-build-upload

Conversation

@shykes

@shykes shykes commented Jun 15, 2013

Copy link
Copy Markdown
Contributor
  • Builder: simplify the upload of the build context. Simply stream a tarball instead of multipart upload with 4 intermediary buffers. Simpler, less memory usage, less disk usage, and faster.
  • Builder: added a regression test for Fix a bug which caused builds to fail if ADD was the first command #895
  • Builder: reorganized unit tests for better code reuse, and to test non-empty contexts

@shykes

shykes commented Jun 15, 2013

Copy link
Copy Markdown
Contributor Author

I'm hitting a bug when combining Tar() + http chunked transfer, closing until I can find it.

@shykes shykes closed this Jun 15, 2013
@shykes shykes reopened this Jun 18, 2013
@shykes

shykes commented Jun 18, 2013

Copy link
Copy Markdown
Contributor Author

So the bug was not in this branch after all (and was separately fixed in #885).

I'm ready for review /cc @creack @vieux

@shykes

shykes commented Jun 18, 2013

Copy link
Copy Markdown
Contributor Author

ping @creack @vieux

@vieux

vieux commented Jun 18, 2013

Copy link
Copy Markdown
Contributor

I see there is an update of APIVERSION, could you add some doc ?

@shykes

shykes commented Jun 19, 2013

Copy link
Copy Markdown
Contributor Author

ping @creack

@creack

creack commented Jun 19, 2013

Copy link
Copy Markdown
Contributor

Couple of minor things:

  • If we remove the unitTestImageName constant, it might be interesting to remove it everywhere.
  • The tests require internet connectivity.
  • This most likely rejects and close Allow separate Dockerfile from context #901.
  • As mentioned by @vieux, the api version changed, so we need to create a docker_remote_api_v1.3.rst file in the docs.

Otherwise, LGTM

@shykes

shykes commented Jun 19, 2013

Copy link
Copy Markdown
Contributor Author

Actually I need to bring back the multipart upload code to keep reverse compatibility with 1.2 clients and daemons... :(

@shykes

shykes commented Jun 19, 2013

Copy link
Copy Markdown
Contributor Author

Closing until that's done.

@shykes shykes closed this Jun 19, 2013
Solomon Hykes added 3 commits June 19, 2013 14:59
… semver, but our API should still be in 0.X versioning, in which case semver allows breaking changes.
@shykes shykes reopened this Jun 19, 2013
@shykes

shykes commented Jun 19, 2013

Copy link
Copy Markdown
Contributor Author

I added documentation for 1.3 API.

I also added a warning for clients connecting to /build for version 1.2 or older. Ideally we would preserve reverse-compatibility with the old build method, but it would be a lot of work for not much benefit, since the API should still be considered unstable anyway.

@shykes

shykes commented Jun 19, 2013

Copy link
Copy Markdown
Contributor Author

Therefore I am ready for review @vieux @creack

@vieux vieux mentioned this pull request Jun 20, 2013
@vieux

vieux commented Jun 20, 2013

Copy link
Copy Markdown
Contributor

The whole docs/sources/api/docker_remote_api.rst needs to be updated (current version 1.3, 1.3 section with what's new)

@vieux

vieux commented Jun 20, 2013

Copy link
Copy Markdown
Contributor

Otherwise, LGTM

@shykes

shykes commented Jun 20, 2013

Copy link
Copy Markdown
Contributor Author

FIxed references to 1.3.

I can't find the "what's new" section in the other versions of the API docs. Should I just add one?

Conflicts:
	api.go
	builder_client.go
	commands.go
@creack

creack commented Jun 20, 2013

Copy link
Copy Markdown
Contributor

There is an issue with the merge. commands.go l196, cli.host and cli.port does not exist anymore, it has been replaced by cli.addr.
The TestBuild fails. There is an issue with ADD, for some reason ADD foo /usr/lib/bla/bar puts foo in /usr/lib/bla.

@shykes

shykes commented Jun 21, 2013

Copy link
Copy Markdown
Contributor Author

Yeah I'm aware of that ADD problem, fix underway. (But I think it's
orthogonal to #900, I believe it's already present in master).

On Thu, Jun 20, 2013 at 4:24 PM, Guillaume J. Charmes <
notifications@github.com> wrote:

There is an issue with the merge. commands.go l196, cli.host and cli.port
does not exist anymore, it has been replaced by cli.addr.
The TestBuild fails. There is an issue with ADD, for some reason ADD foo
/usr/lib/bla/bar puts foo in /usr/lib/bla.


Reply to this email directly or view it on GitHubhttps://github.com//pull/900#issuecomment-19790106
.

Solomon Hykes added 2 commits June 20, 2013 20:25
@shykes

shykes commented Jun 21, 2013

Copy link
Copy Markdown
Contributor Author

Ok, we should be good. I have fixed ADD behavior and added tons of regression tests. It's also much easier to add new tests for builds with non-empty contexts.

@shykes

shykes commented Jun 21, 2013

Copy link
Copy Markdown
Contributor Author

Ping @victor @creack

@vieux

vieux commented Jun 21, 2013

Copy link
Copy Markdown
Contributor

Small merge issue around line 200 in commands.go

diff --git a/commands.go b/commands.go
index 5dffdcc..8e057c4 100644
--- a/commands.go
+++ b/commands.go
@@ -197,7 +197,14 @@ func (cli *DockerCli) CmdBuild(args ...string) error {
                return err
        }
        req.Header.Set("Content-Type", "application/tar")
-       resp, err := http.DefaultClient.Do(req)
+       dial, err := net.Dial(cli.proto, cli.addr)
+       if err != nil {
+               return err
+       }
+       clientconn := httputil.NewClientConn(dial, nil)
+       resp, err := clientconn.Do(req)
+       defer clientconn.Close()
        if err != nil {
                return err
        }

And you still need to update http://docs.docker.io/en/latest/api/docker_remote_api/

@vieux

vieux commented Jun 21, 2013

Copy link
Copy Markdown
Contributor

Also, nothing new (same issue in master) but when your Dockerfile doesn't contain any CMD,
the builded image can be run without any command and will execute the last RUN command in the from the Dockerfile.
I think it should be fixed by this PR.

@shykes

shykes commented Jun 21, 2013

Copy link
Copy Markdown
Contributor Author

About the api doc update - what needs to be updated exactly?

@solomonstre
@getdocker

On Fri, Jun 21, 2013 at 5:54 AM, Victor Vieux notifications@github.com
wrote:

Small merge issue around line 200 in commands.go
diff --git a/commands.go b/commands.go
index 5dffdcc..8e057c4 100644
--- a/commands.go
+++ b/commands.go
@@ -197,7 +197,14 @@ func (cli *DockerCli) CmdBuild(args ...string) error {
return err
}
req.Header.Set("Content-Type", "application/tar")
- resp, err := http.DefaultClient.Do(req)
+ dial, err := net.Dial(cli.proto, cli.addr)
+ if err != nil {
+ return err
+ }
+ clientconn := httputil.NewClientConn(dial, nil)
+ resp, err := clientconn.Do(req)
+ defer clientconn.Close()
if err != nil {
return err
}

And you still need to update http://docs.docker.io/en/latest/api/docker_remote_api/

Reply to this email directly or view it on GitHub:
#900 (comment)

@vieux

vieux commented Jun 21, 2013

Copy link
Copy Markdown
Contributor

in docker_remote_api.rst

you should change The current verson of the API is 1.2 to 1.3 and add the 1.4 section:

:doc:`docker_remote_api_v1.3`
*****************************

What's new
----------

.....

@creack creack merged commit c1a5318 into master Jun 21, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants