osbuilder: add dracut build method by marcov · Pull Request #312 · kata-containers/osbuilder

marcov · 2019-06-12T17:04:09Z

Add the option to build image and initrd using dracut.

Fixes: #311

Suggested-by: Stefan Hajnoczi stefanha@redhat.com
Signed-off-by: Marco Vedovati mvedovati@suse.com

Here's some explanations (this may be later moved in a README).

Changes required

To support the dracut use case, I had to do two set of changes.

The first is to refactor the rootfs-builder script so that the rootfs build step and provisioning step are done independently.
This is to reuse the part of rootfs-builder that adds to a generic rootfs Kata specific components, like the kata-agent installation.
In case of dracut, the provisioning step is done BEFORE the initrd/image is built, and the result is passed to dracut as an overlay to include in the rootfs.

The other change is about the main Makefile. Now when no DISTRO= is specified by the user, the default build target is a dracut initrd / image.

When building an initrd, a Kata-specific overlay is first created using the rootfs-builder.
The dracut initrd is left uncompressed to have faster startup performances, that's however not required.

A dracut image is derived from an extracted dracut initrd. The difference compared to the initrd is that kernel modules needs to be loaded at startup using /etc/modules-load.d systemd method, so that directory needs to include a file with the kernel modules names in dracut.conf.d/10-drivers.conf (these is not required in case of an initrd as kernel modules are loaded automatically).

The extracted rootfs is threated as a normal rootfs directory handled with image-builder.

Example usage

The dracut build method is set up to be paired with a generic / kvm kernel provided by a distribution. These types of kernels require kernel modules like 9p or virtio_xxx to be loaded at startup to be used with Kata.

To obtain this, users will need to:

Install the kernel modules on their host in /lib/modules/VERSION-FLAVOR
specify what kernel modules are to load in rootfs/dracut.conf.d/10-drivers.conf
Building using make initrd|image DRACUT_KVERSION=VERSION-FLAVOR

If a monolithic kernel is used, as the default Kata kernel, these flow is optional and no drivers / DRACUT_KVERSION needs to be specified.

What's missing

Updating documentation
Building in a Docker container
Add tests for dracut
Making sure essential packages like iptables and chrony are installed

jodh-intel

Thanks for raising @marcov. I've got a query on the change.
But also:

Please can you update the appropriate README files to mention dracut.
How are we going to test this?

Can you find a way to add dracut support to tests/test_images.sh?

Makefile

rootfs-builder/rootfs.sh

marcov · 2019-06-18T16:20:26Z

Please can you update the appropriate README files to mention dracut.

How are we going to test this?
Can you find a way to add dracut support to tests/test_images.sh?

I will do that. Given that tests are run inside containers, it makes sense to do the same. The number of builds needed will double though (one dracut build for each distro).

jodh-intel · 2019-06-19T09:48:31Z

The number of builds needed will double though (one dracut build for each distro).

🤞 this is ok as we're testing in parallel 🙏

marcov · 2019-07-17T17:25:31Z

I have finalized what missing to make this PR (hopefully) merge-ready.

Can you please have a review or leave any feeedback?
/cc @kata-containers/builder @devimc @jcvenegas @grahamwhaley @jodh-intel

Add the use case of provisioning an existing rootfs directory with the components / configurations needed to generate a Kata compatible images. This supports use cases such as using a rootfs built outside of osbuilder, and providing a overlay for dracut built initrds. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

.ci/setup.sh

marcov · 2019-07-18T07:54:35Z

@jodh-intel
spellcheck fails on Travis with:

INFO: Spell check successful for file: 'README.md'
INFO: Spell checking file 'rootfs-builder/README.md'
WARNING: Word ''dracut'': did you mean one of the following?: Dracula, underact, distract
WARNING: Word 'Customise': did you mean one of the following?: Customize, Decustomised, Customhouse, Customer, Custom, Taxidermist, Costumer
WARNING: Word 'Customise': did you mean one of the following?: Customize, Decustomised, Customhouse, Customer, Custom, Taxidermist, Costumer
ERROR: Spell check failed for file: 'rootfs-builder/README.md'
ERROR: spell check failed for document rootfs-builder/README.md
The command ".ci/setup.sh" failed and exited with 1 during

But "dracut" is in the words list here:
https://github.com/kata-containers/tests/blob/659d8de512a592153fb3a308abdda98249d2e5ad/cmd/check-spelling/data/projects.txt#L18

jodh-intel · 2019-07-18T08:11:20Z

@marcov - if you look at that error carefully, it's because the word has been quoted using apostrophes - that is now "wrong" - we should only quote using double quotes to avoid confusing the spell checker (I've updated all docs across all repos for this btw ;)

See:

https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#the-apostrophe

The fix: just use the word without quotes since, as you say, it is already in the dictionary. I'm not sure why it's complaining about "customise" since that works fine for me on an Ubuntu 16.04 system, as Travis is using...?!?

jodh-intel

Thanks @marcov. A few comments.

jodh-intel · 2019-07-18T10:48:40Z

scripts/lib.sh

+	[ -n "${http_proxy:-}" ] && curlOptions+=("-x ${http_proxy:-}")
+	readonly install_go="
+RUN cd /tmp ; curl ${curlOptions[@]} https://storage.googleapis.com/golang/go${GO_VERSION}.linux-${goarch}.tar.gz
+RUN tar -C /usr/ -xzf /tmp/go${GO_VERSION}.linux-${goarch}.tar.gz


Nit: /usr/local/?

This commit just moved 2 functions from rootfs.sh to lib.sh. If /usr/local is preferred, it's worth creating a new issue.

README.md

rootfs-builder/README.md

README.md

marcov · 2019-07-19T14:06:42Z

/test

Pennyzct · 2019-07-25T06:03:50Z

Hi @marcov Sorry for the red cross. 😢
@grahamwhaley I think that $GOPATH has been set to the correct place $WORKPLACE/go. We log the env everytime. I just pasted partial output here:

+ echo 'Our ENV looks like:'
Our ENV looks like:
+ env
ghprbPullId=312
ghprbPullTitle=osbuilder: add dracut build method
GIT_COMMITTER_NAME=katacontainersbot
GIT_COMMIT=58f42d6dab72e7e32886cf5a2d212ce407f08e4f
……….
USER=jenkins
GOPATH=/home/jenkins/workspace/kata-containers-osbuilder-ARM-18.04-PR/go
BUILD_NUMBER=50
ghprbTargetBranch=master

It looked to me that something went wrong in static check.

file=rootfs/usr/share/doc/gnutls/CONTRIBUTING.md name=kata-check-markdown pid=29929 source=check-markdown version=0.0.1
time="2019-07-22T14:18:41.663650906Z" level=fatal msg="open rootfs/usr/share/doc/gnutls/CONTRIBUTING.md: no such file or directory" commit=65a0c53d0aefee0262ef27738e71d02ae549e837 name=kata-check-markdown pid=29929 source=check-markdown version=0.0.1

And it confused me a bit, the checking target, rootfs/usr/share/doc/gnutls/CONTRIBUTING.md. It's not included in this PR. 😵

jcvenegas · 2019-07-25T14:16:19Z

@Pennyzct any chance we are missing a clenup in the server?

marcov · 2019-07-25T19:00:34Z

@grahamwhaley I think that $GOPATH has been set to the correct place $WORKPLACE/go. ```

@grahamwhaley who should nuke the $WORKSPACE dir? Any reason why this is not happening here?

GabyCT · 2019-07-25T19:53:25Z

This kata-containers/tests#1855 corrects the issue that install_firecracker.sh does not fail when we run it multiple times

grahamwhaley · 2019-07-26T08:17:07Z

@grahamwhaley I think that $GOPATH has been set to the correct place $WORKPLACE/go. ```

@grahamwhaley who should nuke the $WORKSPACE dir? Any reason why this is not happening here?

It is a tickbox option on the jenkins job - it can clean up (delete) the workspace either before or after a run. We tend to tick 'before new run', as that then leaves the workspace around in case we need to get more info or grub around on the previously failed run.

Sooo @marcov - with that fix on the fc installation, do we think the CI is fixed, and should we hand-re-trigger the ARM one then?

marcov · 2019-07-26T09:18:55Z

The detection of the storage driver in docker is now fixed for the fc job. It's failing to start the test container, but that's another issue:

11:10:56 INFO: running: /tmp/jenkins/workspace/kata-containers-osbuilder-ubuntu-1804-firecracker-PR/go/src/github.com/kata-containers/osbuilder/tests/../../tests/cmd/kata-manager/kata-manager.sh configure-image /tmp/osbuilder-test.2Y3awPs/images/kata-containers-image-centos.img
11:10:56 Unable to find image 'busybox:latest' locally
11:10:56 latest: Pulling from library/busybox
11:10:56 ee153a04d683: Pulling fs layer
11:10:56 ee153a04d683: Download complete
11:10:57 ee153a04d683: Pull complete
11:10:57 Digest: sha256:9f1003c480699be56815db0f8146ad2e22efea85129b5b5983d0e0fb52d9ab70
11:10:57 Status: Downloaded newer image for busybox:latest
11:11:32 docker: Error response from daemon: OCI runtime create failed: Failed to check if grpc server is working: context deadline exceeded: unknown.
11:11:32 INFO: ERROR: test failed
11:11:32 INFO: rootfs:
11:11:32 total 24
11:11:32 dr-xr-xr-x 17 root root 4096 Jul 26 09:00 centos_rootfs
11:11:32 drwxr-xr-x 17 root root 4096 Jul 26 08:29 clearlinux_rootfs
11:11:32 drwxr-xr-x 21 root root 4096 Jul 26 09:09 debian_rootfs
11:11:32 dr-xr-xr-x 17 root root 4096 Jul 26 09:01 fedora_rootfs
11:11:32 drwxr-xr-x 22 root root 4096 Jul 26 09:05 suse_rootfs
11:11:32 drwxr-xr-x 21 root root 4096 Jul 26 09:08 ubuntu_rootfs
11:11:32 INFO: images:
11:11:32 total 524292
11:11:32 -rw-r--r-- 1 root root 536870912 Jul 26 09:10 kata-containers-image-centos.img

marcov · 2019-07-26T09:23:37Z

About the ARM failure instead, @grahamwhaley and me identified two problems:

.ci/setup.sh runs .ci/install_kata_image.sh, and that leaves the ./rootfs tree at the osbuilder path. It would be better to clean all build artifacts after this, so that the tests could start from a clean clone. WDYT @chavafg ?
The spell check should only run on files versioned by Git. WDYT @jodh-intel?

@Pennyzct, in the end everything seems to be fine with the ARM machine setup. Thank you for checking!

jodh-intel · 2019-07-26T09:49:59Z

@marcov - agreed - it wouldn't be reasonable to "fail" a PR due to doc files typos that are not part of this project ;)

chavafg · 2019-07-26T14:06:45Z

@marcov did you try by removing one of the .ci/setup.sh from the PR? #312 (review)

I think that may be causing the issues as the second time you are running the setup, there are already artifacts in place.

marcov · 2019-07-26T16:04:10Z

/test

marcov · 2019-07-26T16:32:10Z

ARM ./rootfs folder is still there 😭

ERROR: Document rootfs/usr/share/doc/libidn2/README.md is not referenced
Build step 'Execute shell' marked build as failure

Pennyzct · 2019-07-29T02:32:49Z

Hi~ @marcov Let me log into this ARM CI node, and clean the env for you. ;). Let's see if the error is also reproduceable~~

Pennyzct · 2019-07-29T05:26:40Z

Hi~ @marcov I have done the clean-up and rebuild the ARM CI. let's wait and see. ;)

jodh-intel

Thanks @marcov - a few doc nits but once resolved I'm happy to approve.

jodh-intel · 2019-07-29T10:14:31Z

README.md

 The top-level `Makefile` contains an example of how to use the available components.

-By default, components will run on the host system. However, some components
+Two build methods are available, "distro" and "dracut".


Could you put these terms in backticks? distro and dracut as they are arguments.

jodh-intel · 2019-07-29T10:15:18Z

README.md

+to generate an initrd or an image. This is what happens in details:
+1. A rootfs is generated based on the specified target distribution.
+2. The rootfs is provisioned with Kata-specific components and configuration files.
+2. The rootfs is used as a base to generate an initrd or an image.


Although it appears to render correctly this "double 2" is confusing. Please either number sequentially, or use the magic 1. for all lines to have GitHub auto-number the bullets for you.

jodh-intel · 2019-07-29T10:16:40Z

README.md


+#### dracut based rootfs
+
+> Note: the dracut build method does not need a rootfs as a base for an image or initrd.


Please make the note bold as specified in https://github.com/kata-containers/documentation/blob/master/Documentation-Requirements.md#notes and capitalise the first word:

Note: The dracut ...

Same comment for notes below.

jodh-intel · 2019-07-29T10:18:43Z

rootfs-builder/README.md

-To list the supported distributions, run:
+### Extra features
+#### Supported distributions list
+Supported distributions can be listed with:


This should be more direct to keep @klynnrif happy I think:

List supported distributions by running the following:

jodh-intel · 2019-07-29T10:19:15Z

rootfs-builder/README.md

+```
+
+#### Generate Kata specific files
+`rootfs.sh` can be used to populate a directory with only Kata specific files and


The rootfs.sh script can ...

jodh-intel · 2019-07-29T10:20:08Z

rootfs-builder/README.md

 ```
-$ ./rootfs.sh -h
+$ mkdir kata-overlay
+$ ./rootfs.sh -r `pwd`/kata-overlay


Nit: I'd try to avoid backticks in commands as they can be hard to see. Maybe use the variable instead:

$ ./rootfs.sh -r "$PWD/kata-overlay"

jodh-intel · 2019-07-29T10:20:25Z

rootfs-builder/README.md

 rootfs directory to contain `/sbin/init` and `/sbin/kata-agent` binaries.

-### Optional - Customise the rootfs
+### Optional - Customize the rootfs


No comment. Let the Holy Spelling Wars not begin... 😄

The spell checker is making the laws now 😆

Or should that be lawz? 😄

Add the option to build image and initrd using dracut. Fixes: kata-containers#311 Suggested-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Marco Vedovati <mvedovati@suse.com>

marcov · 2019-07-29T13:23:55Z

Thanks @jodh-intel, I agree on all your comments and I have addressed them!

Hopefully with this PR ARM builds should not fail anymore (BTW thanks @Pennyzct for trying with a clean build).

/test

Build dracut initrd and images inside a reference container. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

Use .ci/setup.sh from tests repo, to make sure spell check packages are correctly installed for the OS under test. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

Ubuntu bionic is a more recent LTS than trusty. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

Distros with systemd / agent as init are not correctly enumerated. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

marcov · 2019-07-29T15:18:34Z

Forgot to disable the dracut initrd test for firecracker CI job, otherwise it should finally be all set

/test

jodh-intel

Thanks @marcov.

lgtm

jcvenegas · 2019-07-29T17:01:26Z

all green! awesome! I taking a look in a few minutes.

marcov added rfc Requires input from the team wip Work in Progress (PR incomplete - needs more work or rework) labels Jun 12, 2019

marcov force-pushed the dracut branch 3 times, most recently from 5d1d619 to 5ca1b8b Compare June 12, 2019 17:53

jodh-intel reviewed Jun 18, 2019

View reviewed changes

Makefile Outdated Show resolved Hide resolved

rootfs-builder/rootfs.sh Outdated Show resolved Hide resolved

marcov force-pushed the dracut branch 3 times, most recently from 7480834 to 1547733 Compare July 17, 2019 17:21

marcov removed rfc Requires input from the team wip Work in Progress (PR incomplete - needs more work or rework) labels Jul 17, 2019

marcov force-pushed the dracut branch 3 times, most recently from 2a238f8 to 3077f65 Compare July 17, 2019 17:44

marcov force-pushed the dracut branch 2 times, most recently from 0452f0a to c67855a Compare July 17, 2019 17:53

marcov commented Jul 17, 2019

View reviewed changes

.ci/setup.sh Outdated Show resolved Hide resolved

grahamwhaley requested a review from jcvenegas July 18, 2019 09:13

marcov force-pushed the dracut branch 3 times, most recently from a7eed70 to bcd3280 Compare July 18, 2019 10:28

jodh-intel reviewed Jul 18, 2019

View reviewed changes

marcov force-pushed the dracut branch from bcd3280 to 2a2a435 Compare July 19, 2019 09:37

marcov force-pushed the dracut branch from f17b5ee to 4cc8ffe Compare July 26, 2019 15:04

jodh-intel reviewed Jul 29, 2019

View reviewed changes

osbuilder: add dracut build method

2f55017

Add the option to build image and initrd using dracut. Fixes: kata-containers#311 Suggested-by: Stefan Hajnoczi <stefanha@redhat.com> Signed-off-by: Marco Vedovati <mvedovati@suse.com>

marcov force-pushed the dracut branch from 4cc8ffe to 5c1e07c Compare July 29, 2019 13:21

marcov force-pushed the dracut branch from 5c1e07c to 60523f6 Compare July 29, 2019 15:09

marcov added 4 commits July 29, 2019 17:10

tests: add dracut initrd and image tests

1e5746a

Build dracut initrd and images inside a reference container. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

ci: use .ci/setup.sh from tests repo

a5a3343

Use .ci/setup.sh from tests repo, to make sure spell check packages are correctly installed for the OS under test. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

travis: use Ubuntu bionic distro version

317bfba

Ubuntu bionic is a more recent LTS than trusty. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

tests: fix background job failure detection

934422f

Distros with systemd / agent as init are not correctly enumerated. Signed-off-by: Marco Vedovati <mvedovati@suse.com>

marcov force-pushed the dracut branch from 60523f6 to 934422f Compare July 29, 2019 15:10

jodh-intel approved these changes Jul 29, 2019

View reviewed changes

jcvenegas approved these changes Jul 29, 2019

View reviewed changes

jcvenegas merged commit 73cee17 into kata-containers:master Jul 29, 2019

chavafg mentioned this pull request Aug 1, 2019

chrony-service: re-patch PR#265 #339

Merged


		#### dracut based rootfs

		> Note: the dracut build method does not need a rootfs as a base for an image or initrd.

Conversation

marcov commented Jun 12, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changes required

Example usage

What's missing

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

marcov commented Jun 18, 2019

Uh oh!

jodh-intel commented Jun 19, 2019

Uh oh!

marcov commented Jul 17, 2019

Uh oh!

Uh oh!

marcov commented Jul 18, 2019

Uh oh!

jodh-intel commented Jul 18, 2019

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

marcov commented Jul 19, 2019

Uh oh!

Pennyzct commented Jul 25, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jcvenegas commented Jul 25, 2019

Uh oh!

marcov commented Jul 25, 2019

Uh oh!

GabyCT commented Jul 25, 2019

Uh oh!

grahamwhaley commented Jul 26, 2019

Uh oh!

marcov commented Jul 26, 2019

Uh oh!

marcov commented Jul 26, 2019

Uh oh!

jodh-intel commented Jul 26, 2019

Uh oh!

chavafg commented Jul 26, 2019

Uh oh!

marcov commented Jul 26, 2019

Uh oh!

marcov commented Jul 26, 2019

Uh oh!

Pennyzct commented Jul 29, 2019

Uh oh!

Pennyzct commented Jul 29, 2019

Uh oh!

jodh-intel left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

marcov commented Jun 12, 2019 •

edited

Loading

Pennyzct commented Jul 25, 2019 •

edited

Loading