the cmd() function has been removed from the test utils so the tests will need to be updated

@jfrazelle, thanks for the heads up. I have updated the tests.

I presume that ENV BANANA asd will over-ride the build env value, and that I can persist the build-env values by doing something like RUN export BANANA=$BUILDVAR

All up, nice! - that would allow me to do ...

docker build -e HTTP_PROXY=http://10.10.10.2:4342 -t build ., and for others to add the passwords to access their closed binaries.

which would be useful to add to the apt-cacher-ng article.

In our case we are looking at using Docker to build images of our Web services, which need to obtain JARs from a private (but Internet-accessible) Maven (Artifactory) repository at build time. This PR would allow us to set our auth credentials for that Maven repo at build time, using environment variables which only exist at build time, which would be very useful to us!

Additionally, if the Docker Hub Web UI was extended with a little GUI to set these build-time variables, that would allow us to convert manual builds which our devs must currently run on their boxes to automated builds.

@SvenDowideit

I presume that ENV BANANA asd will over-ride the build env value,

Yes, this PR will allow build env value for BANANA to be overriden by the value provided by ENV statement ('asd' in this case).

and that I can persist the build-env values by doing something like RUN export
BANANA=$BUILDVAR

I am not sure if I understand this presumption. Orthogonal to changes in this PR, I think issuing 'export' during a RUN will not persist the exported environment in the final container itself as it will just be run as a shell command in the context of that RUN statement. And this PR doesn't change anything around that behavior. Am I missing something?

that would allow me to do ...
docker build -e HTTP_PROXY=http://10.10.10.2:4342 -t build ., and for others to add the passwords to access their closed binaries.
which would be useful to add to the apt-cacher-ng article.

Yes, I think this should be addressed by this PR. Were you planning to use persisted environment with 'RUN export' to achieve this?

@themasterchef
Yes, this PR shall address the requirement of passing 'auth' credentials at build time without persisting them.

I have mostly worked with cli interface till now and haven't gotten to familiarizing myself with Docker Hub Web UI. So I just enhanced the cli to start making use of this feature. But I could definitely look into enhancing the web UI as well. Is it ok, if I look into it as a separate issue/PR?

@mapuri I was wondering mostly about the intended function - and I like it - it would be worth adding a little more of what you've answered here into the documentation.

please also note that the cli.md is the primary docs for commands, and the man pages are shorter summaries.

wrt Hub UI - its not open source but your work will enable things that @themasterchef, I and others would like :)

next up - @shykes @crosbymichael ? design review

@SvenDowideit thanks for clarifying and pointers on documentation!

I have updated the cli.md to document more details on the changes. I have also updated docker_remote_api.md and docker_remote_api_v1.16.md to document the optional header, introduced to pass the variables to builder. The hub UI and other remote clients will form and pass this header in order to use this feature. I wasn't sure if this change demands bumping up the API version though, so I just updated v1.16 doc file for now.

Docs LGTM - @fredlf @jamtur01

Looking good, do you have an estimate of when this will be making its way into master? We've just hit the point today where this is feature a really big deal for us.

As there is NO way to conditionally execute Dockerfile instruction during build, how could we address that @mapuri ???

Following shall work to execute commands conditionally, in absence of a Dockerfile native conditional primitive. It does create a intermediate container but I think has same effect as you are looking for. WDYT?

RUN if [ "$HTTP_PROXY" != "" ]; then npm config set proxy $HTTP_PROXY; fi

@mapuri Good News! It worked! It has been a long time since I've been attempting to build from those Cloud hosts and wow, this is the first time!!! I was building images locally on my desktop and pushing to our internal Registry.

Problem solved!

The problem I encountered in regards to github connectivity was related to Node.js' npm settings angular/angular-phonecat#141 (comment)... It was trying to git clone with the "git://" protocol... So, here's the updated Dockerfile as a reference for others to try:

# Pull base image.
FROM node:0.12

RUN echo $HTTP_PROXY
RUN npm config set proxy $HTTP_PROXY && git config --global url."https://".insteadOf git://

COPY package.json /tmp/package.json

RUN if [ "$HTTP_PROXY" != "" ]; then npm config set proxy $HTTP_PROXY && npm config list; fi

RUN cd /tmp && npm install -d

# Node base will default the command to `node server.js`.

# Expose port.
EXPOSE 3000

Docker build output

I grabbed my $HTTP_PROXY settings from the environment so the execution below assumes they are set. From what I can say, this approach is acceptable in the client point of view... Just like any other parameter to run, build also calls for this feature in as a natural requirement of builds in isolated environments.

The output of building the Dockerfile above is as follows:

# root at pppdc9prd8wu in /tmp/stackedit on git:master x [15:42:12]
$ docker build --no-cache=true --build-env http_proxy=$http_proxy --build-env https_proxy=$https_proxy --build-env no_proxy=$no_proxy --build-env HTTP_PROXY=$HTTP_PROXY --build-env HTTPS_PROXY=$HTTPS_PROXY --build-env NO_PROXY=$NO_PROXY .
Sending build context to Docker daemon 40.85 MB
Step 0 : FROM node:0.12
 ---> 20a32f7a591c
Step 1 : RUN echo $HTTP_PROXY
 ---> Running in f39030818bf2
http://qypprdproxy02.ie.intuit.net:80
 ---> 70176cd82684
Removing intermediate container f39030818bf2
Step 2 : RUN npm config set proxy $HTTP_PROXY && git config --global url."https://".insteadOf git://
 ---> Running in 4ccb1450c407
 ---> bbe373ebf241
Removing intermediate container 4ccb1450c407
Step 3 : COPY package.json /tmp/package.json
 ---> cc7f700a0a7b
Removing intermediate container c6889dcb7095
Step 4 : RUN if [ "$HTTP_PROXY" != "" ]; then npm config set proxy $HTTP_PROXY && npm config list; fi
 ---> Running in 7ddb2bfa909a
; cli configs
user-agent = "npm/2.11.3 node/v0.12.5 linux x64"

; userconfig /root/.npmrc
proxy = "http://qypprdproxy02.ie.intuit.net:80/"

; node bin location = /usr/local/bin/node
; cwd = /
; HOME = /root
; 'npm config ls -l' to show all defaults.

 ---> 0baab9052e6b
Removing intermediate container 7ddb2bfa909a
Step 5 : RUN cd /tmp && npm install -d
 ---> Running in 162ef5dc781a
npm info it worked if it ends with ok
npm info using npm@2.11.3
npm info using node@v0.12.5
npm info preinstall stackedit@4.3.11
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/request
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/ssh2
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/compression
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/serve-static
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/gulp
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/gulp-requirejs
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/gulp-jshint
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/gulp-uglify
npm info attempt registry request try #1 at 3:42:34 PM
npm http request GET https://registry.npmjs.org/gulp-less
npm info attempt registry request try #1 at 3:42:34 PM
npm info retry fetch attempt 1 at 3:42:36 PM
npm info attempt registry request try #1 at 3:42:36 PM
npm http fetch GET https://registry.npmjs.org/send/-/send-0.13.0.tgz
npm http 200 https://registry.npmjs.org/nopt
npm info build /tmp/node_modules/mime
npm info preinstall gulp-requirejs@0.1.3
npm http 200 https://registry.npmjs.org/update-notifier
npm http 200 https://registry.npmjs.org/minimatch
npm http fetch 200 https://registry.npmjs.org/escape-html/-/escape-html-1.0.2.tgz
npm http fetch 200 https://registry.npmjs.org/on-headers/-/on-headers-1.0.0.tgz
npm http 200 https://registry.npmjs.org/lodash._reescape
npm info attempt registry request try #1 at 3:42:36 PM
npm http request GET https://registry.npmjs.org/orchestrator
npm info attempt registry request try #1 at 3:42:36 PM
npm http request GET https://registry.npmjs.org/pretty-hrtime
npm info attempt registry request try #1 at 3:42:36 PM
npm http request GET https://registry.npmjs.org/semver
npm info attempt registry request try #1 at 3:42:36 PM
npm http request GET https://registry.npmjs.org/tildify
npm info attempt registry request try #1 at 3:42:36 PM
npm http request GET https://registry.npmjs.org/v8flags
...
...
...
└── inquirer@0.8.0 (ansi-regex@1.1.1, figures@1.3.5, mute-stream@0.0.4, through@2.3.8, readline2@0.1.1, chalk@0.5.1, lodash@2.4.2, cli-color@0.3.3, rx@2.5.3)

bower-requirejs@1.1.3 node_modules/bower-requirejs
├── slash@1.0.0
├── object-assign@2.1.1
├── sudo-block@1.2.0 (is-root@1.0.0, is-docker@1.0.0)
├── chalk@1.1.0 (escape-string-regexp@1.0.3, supports-color@2.0.0, ansi-styles@2.1.0, has-ansi@2.0.0, strip-ansi@3.0.0)
├── nopt@3.0.3 (abbrev@1.0.7)
├── requirejs@2.1.18
├── file-utils@0.2.2 (isbinaryfile@2.0.4, rimraf@2.4.1, glob@4.5.3, minimatch@2.0.8, findup-sync@0.2.1, iconv-lite@0.4.11, lodash@2.4.2)
├── update-notifier@0.3.2 (is-npm@1.0.0, string-length@1.0.0, semver-diff@2.0.0, latest-version@1.0.1, configstore@0.3.2)
└── lodash@3.10.0

gulp-less@1.3.9 node_modules/gulp-less
├── convert-source-map@0.4.1
├── lodash.defaults@2.4.1 (lodash._objecttypes@2.4.1, lodash.keys@2.4.1)
├── through2@0.5.1 (xtend@3.0.0, readable-stream@1.0.33)
├── vinyl-sourcemaps-apply@0.1.4 (source-map@0.1.43)
└── less@1.7.5 (graceful-fs@3.0.8, mime@1.2.11, source-map@0.1.43, mkdirp@0.5.1, clean-css@2.2.23)
npm info ok
 ---> a261278405d8
Removing intermediate container 162ef5dc781a
Step 6 : EXPOSE 3000
 ---> Running in 463e0c82ae28
 ---> 2773a577834c
Removing intermediate container 463e0c82ae28
Successfully built 2773a577834c

Follow up Questions

@mapuri, the trick with conditions makes me feel we need to address the case either having a conditional instruction or something else. What I'm thinking here is that there could be potentially 2 different images built:

• One from building on an environment with HTTP_PROXY
• One from building without

So, It looks like there are implications on the idempotency of the resulting images on the 2 scenarios... Those settings that I just set were performed for the sake of making the build pass. However, they could potentially affect the normal execution of the container, given the changes.

Has anyone discussed disposable layers? What if I wanna run a conditional Instruction and have it removed in the final Image? Say...

RUN {throwAway, conditional ($HTTP_PROXY != "")} npm config set proxy $HTTP_PROXY

@marcellodesales

Good to know this worked for you!

Has anyone discussed disposable layers? What if I wanna run a conditional Instruction and have it removed in the final Image?
RUN {throwAway, conditional ($HTTP_PROXY != "")} npm config set proxy $HTTP_PROXY

Having the command-line variables available for use in Dockerfile does enable the usecase of parameterized builds as mentioned in some of the comments elsewhere in this PR. I think conditionals in a form that you mentioned shall fall in a similar category. However, I am not sure if there is any open PR or issue for the same.

In addition to the proxy use case, I have one other.

I would like to set tags (TAG) in my Dockerfile for metadata such as the build date. Essentially there are valid uses cases for setting tags in a Dockerfile that are dynamic and change every build, particularly in a CI/CD pipeline, having -e would help. This could also be solved by allowing interpolation within the Dockerfile (I'm pretty sure there is at least one issue open for this) e.g. shelling out to the data command.

Currently at the moment I have to copy the Dockerfile to say Dockerfile.build which is temporary and do a sed replace on a couple of placeholder strings, then build with -f Dockerfile.build.

• 1, I am glad that environment passing is happening in docker build. I been hardcoding the http_proxy in all the internal Dockerfiles...

+1, I don't see a better option to pass in extraneous build information without making one-off conventions that ultimately expose environment data.

Example: Datomic requires each user to authenticate in order to download the datomic zip. This means the datomic image needs a username and password injected on build somehow. You could use a .docker-datomic-credentials file, but then you have to know to populate a .docker-datomic-credentials file (or something else). That is unintuitive (a bunch of files, like .docker-proxy, that all are only relevant for exposing environment variables to docker and nobody knows that unless they read the Dockerfile or parent image Dockerfile).

Eventually it seems you would want a convention like docker-onbuild.environment to somewhat address the intuitiveness concern, which means adding your own COPY onbuild.environment /tmp/docker-onbuild.environment, RUN . /tmp/docker-onbuild.environment && ... and RUN rm /tmp/docker-onbuild.environment commands to expose those environment variables only on container builds.

Then you realize it would be nice to include this file as an argument so you don't have to worry about missing the rm at the end of your Dockerfiles (and not have to source the variables on every RUN). That leads to the ability to have a docker build --env-file onbuild.environment . or docker-build --env "FOO=bar" ..

+1

I'm in much need of this feature to support privately forked repositories.

+1
This would greatly simplify my already generic build process

+1
long awaited options!

+1

+1 the Docker build is totally dependent on the state of the files it's working with, so if those files weren't placed predictably then the build will result in a different output. That argument is a strawman.

Many, many users including myself are using hacks to pipe in information into the build process... we need this PR.

+1

@mapuri The signal to noise ratio in this thread has reach a point where I just cannot keep up. Please come ping me on IRC or send me a mail, and we can discuss the way to move forward into a solution that we will merge. Thank you for your work and time.

Closing this in favor of #14634 .

I will work on adapting the changes in this PR to the spec detailed in #14634. Hoping to address the use cases that this thread brought up.