improve handling of ISO boot media

As discussed in https://github.com/linuxkit/linuxkit/pull/2363 rather than defaulting to root=/dev/sr0 or having the user specify the boot medium, it ought to be possible at least for EFI to detect the boot medium (needs a tiny initramfs) or for EFI and BIOS CDs to use UUID boot although that may be unworkable as the kernel wants a PARTUUID.

As far as I can tell this does work on Windows, it is certainly mounting the rootfs and starting containerd.

Works on Windows? I don't understand?

In the other issue, I wrote:

So... kernel needs PARTUUID, and iso9660 only has UUID. There also is a "relative to a known UUID". Perhaps we can put a PARTUUID on the ESP we create to put in the iso9660, and then relative from there?

Or perhaps when we have a proper hybrid with a GPT we can use it that way.

@deitch by "works on windows" I just tested booting it - EFI ISO is the only format we generally use on Hyper-V. I had some networking issues on my Windows VM so dhcp was hanging but other than that it was fine.

by "works on windows" I just tested booting it

Oh, you mean "Windows with hyper-v as hypervisor works with /dev/sr0". OK, got it.

@justincormack There's potential issue with this method, root=/dev/sr0 is just the kernel command line, not used by the UEFI firmware, so if we have the 'CONFIG_ISO9660_FS=m' in the kernel configuration file(I think we do support kernel modules in LinuxKit), then we will have trouble since the kernel will be panic when mounting the rootfs in /dev/sr0, because the ISO9660 is in the initrd and have no chance to be loaded. I am trying to unify the UEFI ISO image generation for both x86_64 and arm64, we will use GRUB2 as the boot loader to address the issue of #2359 instead of the gummiboot

Dennis if that were correct, how would it boot at all? Yet it does.

We actually don't use gummiboot as a boot loader as much as enriching the EFI kernel stub. It is a single file.

I'd prefer if we avoid grub2...

if we have the 'CONFIG_ISO9660_FS=m' in the kernel configuration

I suggest to not do that if you want ISO boot. I don't think we want to get into building initrds for specific boot devices and sufficient drivers/fs support should be compiled in to boot.

Modern kernels have the ability to be booted directly by efi (see efi-stub.txt). However, this does not make it easy to pass in the command line (it would normally be compiled in) so we further wrap this with the efi stub code from gummiboot.

So ideally we find a similar way for ARM

Dennis if that were correct, how would it boot at all?

That because current ISO9660 fs module is built into the kernel while not as an external kernel module, we can try to change it from CONFIG_ISO9660_FS=y to CONFIG_ISO9660_FS=m, then we can this scenario.

@arm64b we prefer to build in modules that are needed for booting, otehrwise booting becomes a multi stage process that is much slower. Grub is just a cut down version of Linux with the modules built in, it is nicer to just boot directly.

@justincormack @rn OK, it's fine if we want to boot the kernel directly. We can use a very simple grub.cfg to boot the kernel directly and mount the rootfs in the /dev/sr0 if we strictly forbid to build the boot related stuff as external kernel module, the grub.cfg can looks like:

menuentry "LinuxKit ISO EFI boot"
kernel=kernel root=/dev/sr0

Thus we can unify the iso-efi format generation for both x86_64 and arm64, since gummiboot does work on arm64. any thoughts?

@arm64b as said, I would prefer not to use grub at all.

I noticed that the alpine gummiboot package also has the EFI stub https://pkgs.alpinelinux.org/contents?branch=v3.6&name=gummiboot&arch=aarch64&repo=main

can't we then boot arm64 the same way as x86?

@rn I've tried the gummiboot pkg of arm64, from my experience, that's not a EFI stub can be recognized by objcopy & objdump meaning we can't insert the kernel image as a section into that file as x86 does, need to chase down the reason...

we can try to change it from CONFIG_ISO9660_FS=y to CONFIG_ISO9660_FS=m

I think iso9660 is a fairly standard way of booting; we probably should continue to package it in. Why would we move it to a module? Besides, as @justincormack said:

Grub is just a cut down version of Linux with the modules built in

One other side effect: with a single kernel file as EFI direct boot, you can do things like SRTM in Trusted Computing. If you introduce a separate loader like grub, you have to use the measuring one. Standard grub2 does not do that (although Matthew Garett's modified one does). Why take away that ability that works right now?

And, yes, I know, we need to find out about ARM equivalent for tpm.

@deitch Ah, at least in arm64 enterprise ecosystem area, using UEFI + GRUB is a very typical way to boot kernel+initrd, but maybe the design principle of LinuxKit is different with the conventional way in terms of the rational. If we have this agreement to use a tiny EFI application to load the kernel directly like gummiboot does, I will try my best to find a similar solution to fit on arm64, but I will keep an working grub2 solution as my demo for this :)

@deitch I am not familiar with SRTM, and don't know why the Trusted Computing must use a single direct boot kernel with help of gummiboot tool(this tool has been dead for several years, now is systemd-boot). The only reason I can see is the boot speed as @justincormack mentioned, but I am very curious about that do you guys have the micro-benchmark of the booting time between gummiboot and grub2?

don't know why the Trusted Computing must use a single direct boot kernel with help of gummiboot tool

They don't. They do need the UEFI firmware to measure the next step, and the next step must be aware of measuring and measure the next step. In your example, the bootloader must measure into the tpm registers (PCRs) the hashes of each of kernel, command-line, initrd. grub is not natively aware, unless you use the CoreOS one. But, if you use the single file kernel with initrd and command-line bundled in, it is there for free.

There are other ways to get it - including modified grub2 - but this is easier by far.

Ok, thanks for the explanation! Then we will be in trouble on arm64, since objcopy is less functional than on x86, it only supports limited targets. that means we can't generate a PE format-EFI application with those tool chains on arm64 even we pull the antique gummiboot souce code into a native arm64 server, we can't build a PE efi application either.

I will mark it as a challenge on arm64 support

Personally, I am not in love with the gummiboot either, but it works.

Once I get the raw image working on EFI as well, I will put a little focus on a better option too. If grub2 is our only option, so be it. But would be nice to get a better one.

Hi @deitch I'd like to know the Trusted feature for the gummiboot, do we have taken use of it on LinuxKit project now?

do we have taken use of it on LinuxKit project now?

Officially? No, but there are parts of linuxkit that do use it (see some of the tpm elements).

It isn't at all part of gummiboot. gummiboot simply is the fastest solution for it. The alternative is to use some bootloader that will measure the kernel, cmdline and initrd and update the PCRs. To the best of my knowledge, that only is the modified grub2 from CoreOS/Matthew Garrett.

Officially? No, but there are parts of linuxkit that do use it (see some of the tpm elements). It isn't at all part of gummiboot. gummiboot simply is the fastest solution for it. The alternative is to use some bootloader that will measure the kernel, cmdline and initrd and update the PCRs. To the best of my knowledge, that only is the modified grub2 from CoreOS/Matthew Garrett.

Great! I feel the same way, so I am trying to evaluate the solution of GURB2 + Matthew's patch to support those security related features. As far as gummiboot, according to my observation, there's no significant difference between grub and gummiboot in terms of boot speed, I think the highest priority should be the security and have multi-arch (x86 and arm64) support for LinuxKit personally.

with all the ISO boot rework, can this be closed?

Unfortunately not. It still boots from a static /dev/sr0 (for x86_64) and /dev/vda (for arm64). It needs to be redone to have intelligence to determine the boot device and not hard-coded. The easiest probably is a tiny initramfs.

So when building the iso and booting from a usb stick. The kernel does boot but fails to mount the root filesystem. How do I fix this? Can I manually specify /dev/sda as the root filesystem in the yaml config file? Using the sshd example from linuxkit/examples

I don't even understand why I would have to do this though. Other live images just work.

@logicminds I think you should be able to override the rootfs in the config file yes, does that work?

@justincormack I am new to linuxkit. So are you saying something like:

kernel:
  image: linuxkit/kernel:4.17.10
  cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 root=/dev/sda1"

Yes something like that. Not near a computer at the moment, but I think the various output backends append root=... if it’s not specified. For an ISO it would default to /dev/sr0 or some such.

It also just occurred to me that USB might only be enabled as module in the kernel build...

On 8 Aug 2018, at 01:37, Corey Osman [email protected] wrote:

@justincormack I am new to linuxkit. So are you saying something like:

kernel: image: linuxkit/kernel:4.17.10 cmdline: "console=tty0 console=ttyS0 console=ttyAMA0 root=/dev/sda1" — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

Have you considered using grub's search command to find the root device? https://www.gnu.org/software/grub/manual/grub/html_node/search.html