Nova validation checks checks reject working UEFI image

notes for latter

the partion table layonut within the image is not very standard

Model: (file)
Disk /home/smooney/format-inspector-test/flatcar-stable-4081.2.0-kube-v1.30.1.img.raw: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number Start End Size File system Name Flags
 1 2097kB 136MB 134MB fat32 EFI-SYSTEM boot, legacy_boot, esp
 2 136MB 138MB 2097kB BIOS-BOOT bios_grub
 3 138MB 1212MB 1074MB btrfs USR-A
 4 1212MB 2286MB 1074MB USR-B
 6 2286MB 2420MB 134MB btrfs OEM
 7 2420MB 2487MB 67.1MB OEM-CONFIG
 9 2487MB 21.5GB 19.0GB ext4 ROOT

it appear that this image is trying to support both uefi and legacy biors boot mode but the presence of both a bios_grub and legacy_boot seams odd

i don't think it typeical to have both set especially on different portions.

just for refernce here are the partion layouts in some other cloud images

rhel 9.4

[17:14:44]➜ qemu-img convert -f qcow2 rhel-9.4-x86_64-kvm.qcow2 -O raw rhel-9.4-x86_64-kvm.raw
~/format-inspector-test
[17:15:09]➜ parted rhel-9.4-x86_64-kvm.raw
WARNING: You are not superuser. Watch out for permissions.
GNU Parted 3.6
Using /home/smooney/format-inspector-test/rhel-9.4-x86_64-kvm.raw
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: (file)
Disk /home/smooney/format-inspector-test/rhel-9.4-x86_64-kvm.raw: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 2097kB 1049kB bios_grub
 2 2097kB 212MB 210MB fat16 boot, esp
 3 212MB 1286MB 1074MB xfs bls_boot
 4 1286MB 10.7GB 9452MB xfs

centos 10 stream

Disk /home/smooney/format-inspector-test/CentOS-Stream-GenericCloud-10-20241016.0.x86_64.raw: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number Start End Size File system Name Flags
 1 1049kB 2097kB 1049kB bios_grub
 2 2097kB 8391MB 8389MB xfs

ubuntu noble

[17:21:19]➜ parted noble-server-cloudimg-amd64\(1\).raw
WARNING: You are not superuser. Watch out for permissions.
GNU Parted 3.6
Using /home/smooney/format-inspector-test/noble-server-cloudimg-amd64(1).raw
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: (file)
Disk /home/smooney/format-inspector-test/noble-server-cloudimg-amd64(1).raw: 3758MB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
14 1049kB 5243kB 4194kB bios_grub
15 5243kB 116MB 111MB fat32 boot, esp
16 116MB 1074MB 957MB ext4 bls_boot
 1 1075MB 3758MB 2683MB ext4

Rocky 8

GNU Parted 3.6
Using /home/smooney/format-inspector-test/Rocky-8-GenericCloud-Base.latest.x86_64.raw
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) print
Model: (file)
Disk /home/smooney/format-inspector-test/Rocky-8-GenericCloud-Base.latest.x86_64.raw: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags:

Number Start End Size File system Name Flags
 1 1049kB 105MB 104MB fat16 EFI System Partition boot, esp
 2 105MB 1153MB 1049MB xfs primary
 3 1153MB 1158MB 4194kB primary prep
 4 1158MB 1159MB 1049kB primary bios_grub
 5 1160MB 10.7GB 9577MB xfs primary

Fedora ELN

Disk /home/smooney/format-inspector-test/Fedora-ELN-Guest-9.0-0.1707998402.x86_64.raw: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number Start End Size File system Name Flags
 1 1049kB 2097kB 1049kB bios_grub
 2 2097kB 8391MB 8389MB xfs

the flatcar image is significantly more complex then...

ok os repoducing this locally using https://review.opendev.org/c/openstack/oslo.utils/+/937037

the error is

[18:59:29]➜ python3 -m oslo_utils.imageutils.format_inspector ../flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
inspecting file: ../flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
detected file format: gpt
running safety checks...
Safety check mbr on gpt failed because GPT MBR defines invalid extra partitions
FAILED! Safety checks failed: mbr

1/1 failed

and according to

https://wiki.osdev.org/GPT#LBA_0\:_Protective_Master_Boot_Record

""" The UEFI specification requires that the PMBR partition table contain one partition record, with the other three partition records set to zero."""

so i need to look at the detection code in oslo.utils to confirm but I'm 99% sure the flatcar image does not contain a valid PMBR record based on the uefi spec requirements.

as such nova is correctly rejecting the image

it may have been a working image but it does not look like its a valid one.

I'm going to mark this as invalid for nova and add oslo.utils to the bug given this is shared code in the imageutils.

i see a few paths forward.

one close this as invalid and flatcar can make there images conform to the uefi spec.
two add a compatibility flag that relaxes this constraint if opted into
three relax it unconditionally

the concern with 2 and 3 is that if the ovmf firmware in qemu or on real hardware ever enforces the requirement it will break in the future.

option 1 means existing "working" but potentially invlid images will not work on OpenStack.

there are a few things we need to confirm

first does the flatcar image have multiple Partions in the PMBR

as we can see form rocky 8

[18:59:40]❯ python3 -m oslo_utils.imageutils.format_inspector ../Rocky-8-GenericCloud-Base.latest.x86_64.raw
inspecting file: ../Rocky-8-GenericCloud-Base.latest.x86_64.raw
detected file format: gpt
running safety checks...
PASSED!

having multiple partitions is ok, it's listing more then one in the first sector, the Protective Master Boot Record, that is invliad.

second we need to see if we can find a direct refecne to the uefi requirement

third we need to discussion with oslo and the other stakeholder if a compact mot is a viald approach or do we really want to require strict confromance.

ok i have stepped through this code in a debugger

the reported exception is being raised here https://github.com/openstack/oslo.utils/blob/master/oslo_utils/imageutils/format_inspector.py#L1254-L1274

I can confirm that we are detecting two bootable partitions

the first partition

1 2097kB 136MB 134MB fat32 EFI-SYSTEM boot, legacy_boot, esp

has ostype==12

so we add it to valid_partitions but we do not set found_gpt to true

```
            if ostype != 0:
                valid_partitions.append(i)
            if ostype == 0xEE:
                found_gpt = True
                if (starth, starts, startt) != (0x00, 0x02, 0x00):
                    raise SafetyViolation('GPT MBR has invalid start CHS')
                if startlba != 0x00000001:
                    raise SafetyViolation('GPT MBR has invalid start LBA')
```

the second partiotn

2 136MB 138MB 2097kB BIOS-BOOT bios_grub

ostype is 0x33 and we both add it to valid partions and set found_gpt=true

the remain entries have ostype == 0 so
valid_partitions holds [0,1]

that then casses this check to fail

```
    if found_gpt and valid_partitions != [0]:
            raise SafetyViolation('GPT MBR defines invalid extra partitions')
```

this code is intended to enforce the requirement that the first portion in the PMBR has a non 0 entry and that the rest are 0.

i.e. """The UEFI specification requires that the PMBR partition table contain one partition record, with the other three partition records set to zero. """
from https://wiki.osdev.org/GPT#LBA_0\:_Protective_Master_Boot_Record

i belive the relevant part of the uefi spec is

https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#protective-mbr

there we see the portion table entries start at 446
https://github.com/openstack/oslo.utils/blob/79f5ec658e2fee8ab46201a71faaff8d3b67a690/oslo_utils/imageutils/format_inspector.py#L1216

it length is 16*4 bytes and the rescitcion on the other partions being 0 is captured in table 5.3

"""

Array of four MBR partition records. Contains:
• one partition record as defined See Table (below); and
• three partition records each set to zero.

"""

so when we apply that criteria to the oslo.utils implementation

i can confirm that oslo.utils is correctly rejecting the flatcar image as it is not a valid Protective MBR

from the parted output we can also see that the image advertised its self via the disk flags as using pmbr_boot

```
Disk /home/smooney/format-inspector-test/flatcar-stable-4081.2.0-kube-v1.30.1.img.raw: 21.5GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: pmbr_boot

Number Start End Size File system Name Flags
 1 2097kB 136MB 134MB fat32 EFI-SYSTEM boot, legacy_boot, esp
 2 136MB 138MB 2097kB BIOS-BOOT bios_grub
 3 138MB 1212MB 1074MB btrfs USR-A
 4 1212MB 2286MB 1074MB USR-B
 6 2286MB 2420MB 134MB btrfs OEM
 7 2420MB 2487MB 67.1MB OEM-CONFIG
 9 2487MB 21.5GB 19.0GB ext4 ROOT
```

and the from https://www.gnu.org/software/parted/manual/html...

i am using multipel comment ot not loose this detail by the way but i belive i know what the issue is.

if we look at https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#lba-0-format

we are allowed to either have a protective MBR or a legacy mbr

form https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#legacy-master-boot-record-mbr
"""A legacy MBR may be located at LBA 0 (i.e., the first logical block) of the disk if it is not using the GPT disk layout (i.e., if it is using the MBR disk layout)."""

i belvie the the flatcar image is try to provide a modern gpt partion table but writing a legacy mbr
to the first sector which is supported by uefi but is not considered a GPT formated image.

per https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#legacy-master-boot-record-mbr

"""
The following test must be performed to determine if a legacy MBR is valid:

    The Signature must be 0xaa55

    A Partition Record that contains an OSType value of zero or a SizeInLBA value of zero may be ignored.

Otherwise:

    The partition defined by each MBR Partition Record must physically reside on the disk (i.e., not exceed the capacity of the disk).

    Each partition must not overlap with other partitions.
"""

oslo currently does not implement support for validation legacy MBRs

flatcar is not producing an image with legacy mbrs accorded to parted

although file see it as a non texted partion table layout

[20:10:17]➜ file ../flatcar-stable-4081.2.0-kube-v1.30.1.img.raw
../flatcar-stable-4081.2.0-kube-v1.30.1.img.raw: DOS/MBR boot sector
~/format-inspector-test/oslo.utils on  HEAD (3322150) [?] via 🐍 v3.12.7 (.venv)
[20:10:24]➜ file ../Rocky-8-GenericCloud-Base.latest.x86_64.raw
../Rocky-8-GenericCloud-Base.latest.x86_64.raw: DOS/MBR boot sector, extended partition table (last)
~/format-inspector-test/oslo.utils on  HEAD (3322150) [?] via 🐍 v3.12.7 (.venv)
[20:10:37]➜ file ../noble-server-cloudimg-amd64\(1\).raw
../noble-server-cloudimg-amd64(1).raw: DOS/MBR boot sector, extended partition table (last)

so with all that in mind here is my take on this.

the image is not conforment to the gpt and uefi specs.
if we were to extend oslo.utils to validate the first sector as a non-GPT legacy MBR it may pass.

that "should" be safe to allow but it might fail on real hardware or on qemu with a future OVMF release.

so the choice we have is to be strict and require the images to follow the uefi spec
or to be more lax and potentially have non notable images if boot loaders start enforcing this going forward.

Related fix proposed to branch: master
Review: https://review.opendev.org/c/openstack/oslo.utils/+/937215

Hello, the offending commit is this one: https://github.com/openstack/oslo.utils/commit/a0481d5a61da9141e7b60d6d7159f95ab569c211.

Adding magic bit-wise checks to images from more or less documented sources, and making those checks enforcing is, in my opinion, against what OpenStack should do - be a public open source cloud implementation that caters for all the needs.

Agreed that the checks should be there for visibility and warn if one image _might_ not be a bootable one, but in most cases, the underlying implementor like qemu-kvm / libvirt, should be the one that decides that, not the OpenStack code.

Thanks.

unfortunetly we have been explictly tools by qemu that we cannot rely on them to perform security check and the code was introduced as part of mitgiataing a data exflication CVE

the instruction the we were given by libvirt/qemu comunityis is the its the responsibility of the management tools (OpenStack in this case) to ensure the images are safe and that we cannot rely on qemu-img or other tools to be safe to call on untrusted files.

the commit in question is related to https://security.openstack.org/ossa/OSSA-2024-003.html
which was discovered while fixing

https://security.openstack.org/ossa/OSSA-2024-002.html and https://security.openstack.org/ossa/OSSA-2024-001.html

it is very much OpenStack role to validate the content of the image before ever trying to boot

the only question here is whether we treat non-conformatiy to the spec as a potential security issue or
if we relax the check slightly.

For the record, Nova's stance is that we only support using a bootable image with a valid GPT filesystem as the root disk of an instance. That is not a new requirement; the only thing that has changed is that in light of the recent CVEs and the communication form qemu that they expect us to enforce these check we are enforcing it now as a community before passing it to qemu.

i say we are enforcing it as a community because the common code in isolo utils will be used in nova, glance cinder and ironic to enfocne this and we are working to validate image on upload to glace this cycle as a high priority to prevent future cves https://specs.openstack.org/openstack/glance-specs/specs/2025.1/approved/glance/glance-as-defender.html

by the way in case it s not clear why the flatcar image is non conforming

https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#legacy-master-boot-record-mbr

a Legacy Master Boot Record is allowed to have multiple portions in the MBR
provided the The Signature must be 0xaa55, each Partition Record contains an OSType value of zero or a SizeInLBA value of zero, or The partition defined by each MBR Partition Record must physically reside on the disk (i.e., not exceed the capacity of the disk) and Each partition must not overlap with other partitions.

the first partitiion in the is ostype 12 that just meads its fat32 with a particular addressing mode.

the second partition in the flatcar image is ostype 0xEE
https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#os-types

0xEE (i.e., GPT Protective) is used by a protective MBR (see 5.2.2) to define a fake partition covering the entire disk.

so the conflict is you cant have a legacy mbr with multiple non overlaping partitions
and a GPT Protective partions that this required to cover the entire disk at the same time.

https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#mbrdisk-layout-with-legacy-mbr-example

and

https://uefi.org/specs/UEFI/2.10/05_GUID_Partition_Table_Format.html#gpt-disk-layout-with-protective-mbr

are incompatable disk layouts.

what is unknown is could this lead to any security impact.

there are two different actions that need to happen.

first we as an OpenStack community need to decide how to handle non-conforomign images
second we need to report a but to the flatcar community to notify them that there current images are non-conforming so that they can address that.

To me, this is the code doing exactly what it's supposed to do, and that is detect and reject clearly invalid GPT images. The goal of safety-checking raw-converted-from-qcow2 files is specifically to make sure we don't open ourselves (again) to an attacking image that is wrapped in a qcow2 and thus injectable into a system with image conversion. Our ability to say that a qcow2 file we expect to convert into a GPT is actually a GPT when we're done. If you hide a malicious VMDK inside a qcow2 and we do a straight conversion to essentially unwrap the VMDK and then run qemu tools on it, we're toast.

So we can argue about the degree to which we obsess over the correctness of the GPT, although in this case it seems to not be a grey area - the image is invalid according to the spec. It works because the consumer is BIOS, which is extremely basic and unsuspecting of the image, which is why physical disk attacks are still very successful. It's unfortunate that it's a widely-distributed image that violates the spec, but it's still invalid.

The reason we changed the safety check mechanism in the way we did before we merged into oslo was so that the consumer of this library code could (at its own peril) ignore specific safety violations when the context was appropriate (cinder has a specific need here). So we can definitely split the PMBR/GPT and MBR checks to make them distinct, and give nova have a flag to ignore the MBR checks with suitable warnings if we want. However, based on on the operator feedback during the big CVE last year, I tend to expect (or hope) that they would prefer the strictness and just convert/fix the image (which should be easily scriptable) until the distribution can be fixed.

Hello folks,

We will look into fixing this issue in Flatcar itself. The biggest hurdle for us will be to ensure that we don't break upgrades for flatcar users, but this is indeed something that we need to fix on our side.

Sean, thank you for the in depth analysis and for debugging this. Until we fix the image, we will use the existing workaround in the config for testing.

For anyone that stumbles across this bug and needs to boot a current (as of this writing) Flatcar image on OpenStack, and is hitting this issue, the steps to make the image boot properly without setting the disable_deep_image_inspection option are as follows:

qemu-img convert -f qcow2 -O raw flatcar_production_openstack_image.img flatcar_production_openstack_image.raw
parted flatcar_production_openstack_image.raw
set 1 legacy_boot off
quit

You can then upload the flatcar_production_openstack_image.raw to Glance and boot normally. Convert it to qcow2 if you need to, or if you're using Ceph, just leave it as raw. The resulting image should work with both BIOS and EFI.

This change will most likely be a part of the next flatcar stable release.

A fix on the flatcar side has been committed here:

https://github.com/flatcar/scripts/pull/2668

Future releases of Flatcar should work fine.

Change abandoned by "Stephen Finucane <email address hidden>" on branch: master
Review: https://review.opendev.org/c/openstack/oslo.utils/+/937215

Let me shime in.

The issue is *not* only about some distro images that can't be booted. It's also about making an image of something that's already in the cloud, and attempting to boot it again. This will fail, when in the past (before the format_inspector time), it worked. As a result, customers wont be happy, and IMO, rightly.

The goal we're trying to achieve here, is trying to check that we don't get hacked. Being permissive about border-line working GPT images is perfectly OK, as long as there is enough checks so that we make sure the image isn't not hacking the deployment. It's therefore ok if an image isn't strictly respecting the specs, but we check that it is still valid (ie: bootable in UEFI or MBR, and recognized correctly by Qemu, without hacks that would make it possible to read /etc/nova/nova.conf). At least, that's what I am expecting as an operator. I'm not expecting that Nova, Cinder or Glance blocks some images that used to work previously.

Thomas, blocking images that "worked before" is exactly the purpose for format inspector. Before that, we accepted images that were harmful to the entire deployment in obscure ways and now we reject them. Before format inspector, we'd let you upload a perfectly valid qcow2 and call it a raw which was very convenient for users, but very risky for all of our tooling. I can't tell you if or how relaxing the checks for meeting the PMBR spec might introduce other security exposures, but we also didn't see the iceberg on the horizon the last time either.

Especially given that we have prior art of another distro realizing their images were wrong and fixing them to successfully avoid the problem, I feel pretty comfortable keeping this the way it is. If OpenStack is more spec-adhering than your average PC BIOS or other virt platforms, that's a badge I'll proudly wear :)