Skip to content

Add the Moby tool#32693

Closed
justincormack wants to merge 93 commits intomoby:masterfrom
justincormack:moby-tool
Closed

Add the Moby tool#32693
justincormack wants to merge 93 commits intomoby:masterfrom
justincormack:moby-tool

Conversation

@justincormack
Copy link
Contributor

Add the first version of the Moby tool.

See #32691 for context.

This tool is currently being used by LinuxKit https://github.com/linuxkit/linuxkit to assemble full system images from assemblies of containers, and willl be expanded to become a general build tool for building assemblies of components for additional platforms as well.

cute-whales

justincormack and others added 30 commits April 18, 2017 09:37
- terrible code
- lots needs changing
- can build a Moby from a config yaml that boots

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- split out config processing a bit
- just use `capabilities` not `cap-add` and `cap-drop`
- allow use of CAP_ prefix on capabilities, as this is what `runc` uses
- add nginx to example config
- fix bind mounts

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
You can specify a file with contents.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Currently only supports kernel+initrd output but will add the rest
soon.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Note that the EFI ISO is not yet automatically sized, and the
kernel command lines are currently hard coded in the builders.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- remove remainder of editions code
- add a new check container to run tests without Docker
- switch over `make test` to use new command to build tests

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
from:

2017/03/07 09:59:30 Failed to extract kernel image and tarball

to

2017/03/07 10:06:04 Failed to extract kernel image and tarball: Unable to find image 'mobylinux/kernel:7fa748810d7866797fd807a5682d5cb3c9c98111' locally

Signed-off-by: Tycho Andersen <tycho@docker.com>
- based on @riyazdf earlier unmerged Go version https://github.com/riyazdf/moby/blob/891c006d3402698ac883b8063314c7039e6b24d8/alpine/containers/cloud-config/main.go

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Tycho Andersen <tycho@docker.com>
Signed-off-by: Tycho Andersen <tycho@docker.com>
- VHD is uncompressed VHD. Currently hard coded at 1GB, which may need to change. Use `format: vhd`
- GCE is the GCE compressed tarred raw image. Use `format: gce-img` - reserving `gce` for actually
  uploading the image.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- outputs compressed qcow2 image
- 1GB underlying size, may need changing

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
- the image upload uses the cloud API
- currently auth and image creation need the `gcloud` CLI tool.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This requires switching to the dosfstools from alpine:edge since neither the
busybox nor alpine:3.5 dosfstools supports the -C option (in fact alpine:3.5
only has mkfs.fat and not mkfs.vfat).

The 511k slack seems like a lot to me, but 256k was somehow not enough.

Fixes moby#1304.

Signed-off-by: Ian Campbell <ian.campbell@docker.com>
This sets the base name of the built images which otherwise
defaults to the basename of your yaml file. This allows
building different versions easily eg adding git sha to the
output names.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- the `public` option was not previously implemented
- add `replace` only for GCP images which will error otherwise. Only
  recommended for use in development, in production use the `--name` option
  to provide a different name eaxch time. Note only applies to GCP images,
  will document these options properly soon.
- add a `family` option; this allows you to upload many images and the
  user can select the latest using the `family` option instead of a specific
  image.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
- this needs improvements to make it more "platform native", in
  particular GCP supports multiple users and more ssh key mangement
  options.
- at present you can login as root with any platform ssh key
- add support for uts=host and ipc=host
- set the hostname from the metadata as well

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Trying to find the relevant yaml file was an issue as we now support
`--name` and it might be in a different directory, so although it is
a bit verbose outputing a whole file at least it is more consistent.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This was breaking some containers that use it, was a mistake.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
This does not get everything where we want it finally, see moby#1266
nor the optimal way of building, but it gets it out of top level.

Added instructions to build if you have a Go installation.

Not moving `vendor` yet.

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
'moby run' will use the kernel and initrd image produced
by 'moby build' and, on macOS, will run it inside a
hyperkit VM. This assumes that you have a recent version
of Docker for Mac installed as it re-uses the hyperkit
and VPNKit from it.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Removing the left over indirect creates that use the Docker socket
and run in containers not directly.

See moby#1347

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
… VMware Workstation/Fusion

Corrected naming from vmware->vmdk and fixed Makfile

Fixed mistake outputting a vhd instead of a vmdk in output.go

Build vmdk image and added to Docker Hub, corrected link in output.go

Modified directories to confirm to standard mkimage-<imgType>

Signed-off-by: Dan Finneran <dan@thebsdbox.co.uk>
- this removes the use of riddler to extract the rootfs, use code
  we were using for rootfs. riddler now just geenrates the config,
  next stage is to generate this ourselves
- change the naming of the daemons so no longer include number as we
  do not guarantee ordering as they start up simultaneously

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
riyazdf and others added 21 commits April 18, 2017 09:37
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
As suggested by @shykes these are clearer

- onboot for things that are run at boot time to completion
- services for persistent services

Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
Signed-off-by: Dan Finneran <daniel.finneran@gmail.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Signed-off-by: Dave Tucker <dt@docker.com>
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
Signed-off-by: Riyaz Faizullabhoy <riyaz.faizullabhoy@docker.com>
Signed-off-by: Anil Madhavapeddy <anil@docker.com>
Signed-off-by: Dan Finneran <daniel.finneran@gmail.com>
Adds an "access config" with a type of "ONE_TO_ONE_NAT" that
allows an instance to obtain an ephemeral IP address and access the
internet

Signed-off-by: Dave Tucker <dt@docker.com>
This makes gcp behave in a similar way to the qemu backend.
The minimum size on GCP 1GB, whereas qemu uses 256MB.
Without this, the LTP tests fail on GCP.

Signed-off-by: Dave Tucker <dt@docker.com>
Signed-off-by: Dan Finneran <daniel.finneran@gmail.com>
Fix moby#1292

Transform the Yaml struct to JSON and validate it against a JSON schema.

Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>
This uses the Packet.net API and iPXE to boot a Moby host.
There are several enhancements coming soon, such as SSH key
customisation, but this PR is sufficient to boot a host and
then use the web interface to get console access.

The user must currently upload the built artefacts to a public
URL and specify it via --base-url, e.g.:
moby run packet --api-key <key> --project-id <id> \
  --base-url http://recoil.org/~avsm/ipxe --hostname test-moby packet

See moby#1424 moby#1245 for related issues.

Signed-off-by: Anil Madhavapeddy <anil@docker.com>
Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>
Signed-off-by: Thomas Gazagnaire <thomas@gazagnaire.org>
Signed-off-by: Ilya Dmitrichenko <errordeveloper@gmail.com>
Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
GCP defines some "standard" environment variables for project and
zone. Use them for 'moby run gcp'. Change the other environment
variables to follow the same pattern.

Signed-off-by: Rolf Neugebauer <rolf.neugebauer@docker.com>
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
@AkihiroSuda
Copy link
Member

I expect "moby build' to work as if "docker build'

(Is there a plan to implement equivalent of "docker build" in moby?)

"strings"

log "github.com/Sirupsen/logrus"
"github.com/docker/moby/src/initrd"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moby/moby?

@tonistiigi
Copy link
Member

What is the reason for including this tool in the same repo as the engine code(I'm avoiding calling it Moby because it seems that they are both called with the same name)?

This is a CLI utility that depends on a Docker CLI binary(not even API). It uses docker pull to pull images and docker create && docker export to extract squashed images. It also depends on tar and hypervisor launchers already in user's system but doesn't seem to depend on anything from this repo.

It also is very linux/vm centric. Even ignoring the fact that it currently extracts data to initrd, the format itself has terms like kernel, init, onboot that don't really make sense outside the context of creating a vm. I would even call the current tool linuxctl. The engine component, on the other hand, has no relation with these terms. It does not care about what os, kernel starts its process or what other tools are installed in the system. There are probably good use cases for using the engine component as one of the services inside this yml definition but that doesn't really make it different from any other software that can be assembled into a vm like this. If this tool is meant to turn into something else in the future then is there a roadmap for this?

From manifesto, I read that one of the goals is to split up more components. So it is weird that we are starting with putting 2 independent components in the same repo.

@cpuguy83
Copy link
Member

Agreed this shouldn't live here, which I think we discussed F2F.
@justincormack Can you close if that's indeed the conclusion we came up with?

@justincormack
Copy link
Contributor Author

justincormack commented Apr 24, 2017

The agreement we came to was that we would start by moving this to its own repo (and we will remove some of the more VM specific code, and just leave the assembly code), but the aim is that it does go here later, once its assembling stuff properly, and when the rest of the code is removed from here.

It is not intended to continue depending on the Docker CLI, it is mainly going to depend on the containerd api.

@justincormack
Copy link
Contributor Author

This version is out of date.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.