The Docker Vault

(aka The Art of Injecting Ethereal Data into Containers)

This proposal is a docs-based follow up to #6075 and #6697, with some
clarifications and improvements.

Purpose

The purpose for the "Docker vault" is to allow containers (both created in an
intermediate build step or created explicitly) to access data that is considered
to be "secret" while not allowing said data to be stored in images.

Essentially this makes the general purpose of the Docker vault to allow users to
inject arbitrary files into a container context while being assured that said
files will not be leaked into image layers if that container is committed.

This allows for certain use-cases to be accomplished with ease, such as:

• Injecting keys or information required to generate a final image (which is
  allowed to be publicly available), while not storing said keys (which are
  not allowed to be publicly available) in the intermittent layers.
• Injecting keys or other information required for the running of a Docker
  container (such as SSL private keys) when said keys are not allowed to be
  publicly available.

Overview

Data injected into a container using the Docker value will NOT (under any
circumstances) be saved when an image is created from the container. The data
is (to all intents and purposes) ethereal: you can only see it in a container
that has access to it, it cannot be saved into an image.

Containers can be given access to this data, and will be able to access this
data throughout their lifetime (i.e. purging files from the Docker value will
not remove it from running containers).

The data in the vault is conceptually stored in "boxes", where a "box" can be
considered a collection of files that are related in some fashion. Each box is
given a name, and containers are given access to specific boxes. An image can
hint what boxes it requires (just as it can hint the volumes it requires) and
boxes can be "aliased" in a container (you can tell a container that a box named
a is actually a box named b which the image expects).

Box names can consist of any characters except "/" and ":".

It is important to note that vault operations will NEVER affect a running
container, because a running container must remain consistent throughout its
lifetime.

Files in the Docker vault persist across restarts of the Docker daemon.

Containers access files they have been given access to by accessing the files
in /run/vault/<box>/.... Boxes can store directories too. All files and
directories in /run/vault/ are owned by UID and GID 0.

Modifications of any kind to a box inside a container will not be reflected
in the vault's stored boxes. To this end, boxes are mounted ro inside a
container (and remounting them rw will still not allow you to modify
the vault's stored boxes inside a container).

Usage

There are several ways to access the Docker vault functionality:

• docker vault ...: add, query, and delete data from the Docker vault.
• docker create --inject: inject files from the Docker vault into a container.
• docker run --inject: inject files from the Docker vault into a container.
• docker build --inject: inject files from the Docker vault into the
  intermediate containers during the build.
• The Dockerfile syntax explained below.

docker vault

This subcommand has three classes of subcommands, with 6 subcommands in all:

Management (creates and destroys boxes):

• docker vault create [options] <box>...
• docker vault destroy [options] <box>...

Modification (adds files to and removes files from a box):

• docker vault add [options] <box> <file>...
• docker vault remove [options] <box> <file>...

Querying (lists information or accesses data inside boxes or the boxes
themselves):

• docker vault list [options] [<box>...]
• docker vault read [options] <box>[:<file>]...

It also has the following aliases (for the purposes of Unix-like simplicity):

• docker valut ls => docker vault list
• docker vault rm => docker vault remove
• docker vault cat => docker vault read

NOTE: These could just be used as the proper commands and the long-form
ones ignored...

It is very important to note that NONE of the docker vault subcommands
will affect running containers (whether or not they are using a box affected by
the vault operation). Containers use copies of vault data, not references to the
vault data itself -- in order to maintain consistency of a single container's
lifetime.

docker vault create [options] <box>...

This subcommand creates a new box called "<box>" with no files inside it. If
a box with the given name already exists, this command will emit an error and do
nothing.

Options:

• None.

docker vault destroy [options] <box>...

This subcommand obliterates the box called "<box>" and any files stored
within it.

Options:

• -f, --force: ignore errors if the given box name does not exist.

docker vault add [options] <box> <file>...

This subcommand adds the given list of files to the given box. The file names
are preserved when adding the files to the box.

If the box does not exist, the command will emit an error and do nothing.

If one of the paths in the given list does not exist, the command will emit an
error and continue execution.

If the given path points to a symlink, the symlink itself is copied verbatim.

If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).

If the path has some directory components, these will be reflected when the box
is injected into a container. In other words, boxes can store directories.
However, the path will be sanitised, so relative paths (../a/b/c) and absolute
paths /a/b/c will not be reflected. In both cases the paths would be precisely
identical to a/b/c.

If the given path points to a directory, the command will emit an error and
continue execution. If the -r flag is set, then all of the files and
directories in that directory are also added to the box (as if their full paths
were also included in the command). If a directory with the given name already
exists, then the directories are merged.

Options:

• -r, --recursive: recurses directories given, adding all of the contents of
  the directory to the box in addition to the directory itself.

docker vault remove [options] <box> <file>...

This subcommand removes the specified files from the given box.

If the path doesn't exist inside the box, then the command will emit an error
and continue execution.

If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).

If the path is a directory, then the command will emit an error and continue
execution unless the -r flag is set. If the -r flag is set, then the
directory and its contents are recursively removed.

Options:

• -r, --recursive: recurses directories given, removing all of the contents
  of the directories from the box in addition to the directory itself.

docker vault list [options] [<box>...]

This subcommand lists all files and directories stored inside the given boxes.

If no boxes are given, docker vault will list the files in every box stored in
the Docker vault.

If a given box does not exist, the command will emit an error and continue
execution.

Options:

• -b, --boxes: only print the name of each box, not their contents.
• -f FORMAT, --format=FORMAT: formats each line with the given format
  string.
• -r PATTERN, --pattern=PATTERN: only print entries where the file
  paths match the given regular expression.

docker vault read [options] <box>:<file>...

This subcommand reads the contents of each file specified in the command line
and prints them to stdout. No information is printed about which box or file
the data came from.

If a path of box doesn't exist, the command emits an error and continues
execution.

If one of the path components in the path is a symlink, the symlink is followed
as though the box root was the root filesystem (it is scoped to the box).

If a path is a directory, the command emits an error and continue execution
unless the -r flag is set. If the -r flag is set, then the directory is
recursed and all of its contents are

Options:

• None.

docker create [--inject <box>:[<alias>]]...

This option to docker create allows you to inject boxes into a container on
its creation.

If an alias is specified, then the box is injected into /run/vault/<alias>.
Otherwise, the box is injected into /run/vault/<box>.

docker run [--inject <box>:[<alias>]]...

This option to docker run allows you to inject boxes into a container on its
creation.

If an alias is specified, then the box is injected into /run/vault/<alias>.
Otherwise, the box is injected into /run/vault/<box>.

docker build [--inject <box>:[<alias>]]...

This option to docker build allows you to inject boxes into each of the
intermediate build containers during image creation. These boxes will (of
course) not be stored in the resultant image.

If an alias is specified, then the box is injected into /run/vault/<alias>.
Otherwise, the box is injected into /run/vault/<box>.

Dockerfile

Images can hint what boxes they expect in order to run (much like how volume
hinting works). If a hint is not fulfiled, then an empty box is mounted
instead.

Essentially the syntax has two forms (to mirror the VOLUME instruction):

BOX <box>
BOX ["<box>"...]

The first format is a legacy format, only allowing for one box name to be
specified. The second is the newer format, and it accepts a JSON array of
boxes.

Internals

The following documents the following internals:

• Changes to the RESTful API.
• Changes to the container and image information.

RESTful API Changes

Several new endpoints will be added as a result of this functionality:

• PUT /vault/<box>/
• DELETE /vault/<box>/
• PUT /vault/<box>/<path>
• DELETE /vault/<box>/<path>
• GET /vault/
• GET /vault/<box>/
• GET /vault/<box>/<path>

And several modified by this functionality:

• POST /containers/<name>/create
• POST /build

All of which are readibly apparent if you look at the docs for the command-line.

Container and Image Changes

Basically, both the container and image structures need to be updated to store:

• Hinted boxes (images).
• Injected boxes (containers).

Both of which are readily apparent if you look at the docs for the command-line.

/cc @shykes (this is a long one)