Skip to content

sysext: Get verity user certs from given --root=#39991

Merged
yuwata merged 1 commit intosystemd:mainfrom
pothos:user-verity-certs-root
Jan 4, 2026
Merged

sysext: Get verity user certs from given --root=#39991
yuwata merged 1 commit intosystemd:mainfrom
pothos:user-verity-certs-root

Conversation

@pothos
Copy link
Copy Markdown
Contributor

@pothos pothos commented Dec 4, 2025

The verity user certs weren't looked up in the given --root= for systemd-sysext which made it fail to set up extensions with a strict image policy.
Look up verity user certs from inside the --root= when we operate on images in it. The main use case where this matters is when the initrd sets up the extensions for the final system and thus systemd-sysext should do the same thing as it would do in the final system.

@github-actions github-actions bot added machine util-lib tests dissect portable Anything to do with systemd-portable and portablectl and portables sysext please-review PR is ready for (re-)review by a maintainer labels Dec 4, 2025
@bluca
Copy link
Copy Markdown
Member

bluca commented Dec 4, 2025

This doesn't seem right? It's the previouos object that authenticates the next, if the extensions authenticate themselves there's no point...?

@pothos
Copy link
Copy Markdown
Contributor Author

pothos commented Dec 4, 2025

The cert comes from --root= for the extensions in --root=. For example, the initrd sets --root=/sysroot/ and then it will use certs from /usr or /etc to load extensions from /var. The certs under /usr could be coming from the OS vendor and the certs from /etc could be coming from the user. At this point /usr and the rootfs is already trusted. The system itself will also run systemd-sysext refresh again when systemd-sysext.service starts and it will do the same thing again (skipping this is the other PR).

@pothos
Copy link
Copy Markdown
Contributor Author

pothos commented Dec 13, 2025

The tumbleweed CI failure looks strange and I think it only came up after I pushed a rebase:

[  152.334524][ T5597] request_module fs-erofs succeeded, but still no fs?
[  152.337714] TEST-50-DISSECT.sh[5586]: dm resume loop1p1-verity  [ opencount flush readonly securedata ]   [16384] (*1)
[  152.338819] TEST-50-DISSECT.sh[5586]: loop1p1-verity: Stacking NODE_ADD (254,0) 0:6 0640 [trust_udev]
[  152.338991] TEST-50-DISSECT.sh[5586]: loop1p1-verity: Stacking NODE_READ_AHEAD 2048 (flags=1)
[  152.352549] TEST-50-DISSECT.sh[5586]: Udev cookie 0xd4d91f6 (semid 98333) decremented to 1
[  152.353817] TEST-50-DISSECT.sh[5586]: Udev cookie 0xd4d91f6 (semid 98333) waiting for zero
[  152.354415] TEST-50-DISSECT.sh[5586]: Udev cookie 0xd4d91f6 (semid 98333) destroyed
[  152.355602] TEST-50-DISSECT.sh[5586]: loop1p1-verity: Skipping NODE_ADD (254,0) 0:6 0640 [trust_udev]
[  152.355908] TEST-50-DISSECT.sh[5586]: loop1p1-verity: Processing NODE_READ_AHEAD 2048 (flags=1)
[  152.356274] TEST-50-DISSECT.sh[5586]: loop1p1-verity (254:0): read ahead is 2048
[  152.356553] TEST-50-DISSECT.sh[5586]: loop1p1-verity: retaining kernel read ahead of 2048 (requested 2048)
[  152.356863] TEST-50-DISSECT.sh[5586]: dm status loop1p1-verity  [ opencount noflush ]   [16384] (*1)
[  152.357744] TEST-50-DISSECT.sh[5586]: Verity volume loop1p1-verity status is V.
[  152.358138] TEST-50-DISSECT.sh[5586]: Activation of Verity via root hash succeeded.
[  152.359301] TEST-50-DISSECT.sh[5586]: Probed fstype 'erofs' on partition /dev/mapper/loop1p1-verity.
[  152.360371] TEST-50-DISSECT.sh[5586]: Loaded shared library 'libmount.so.1' via dlopen().
[  152.362569] TEST-50-DISSECT.sh[5586]: Successfully forked off '(sd-dissect)' as PID 5597.
[  152.366564] TEST-50-DISSECT.sh[5597]: Failed to open superblock context for 'erofs': No such device
[  152.366888] TEST-50-DISSECT.sh[5597]: Mounting /proc/self/fd/16 (erofs) on /run/systemd/dissect-root (MS_RDONLY|MS_NODEV "")...
[  152.367451] TEST-50-DISSECT.sh[5597]: Failed to mount /proc/self/fd/16 (type erofs) on /run/systemd/dissect-root (MS_RDONLY|MS_NODEV ""): No such device
[  152.367733] TEST-50-DISSECT.sh[5597]: Failed to mount dissected image: No such device

@yuwata yuwata added needs-rebase and removed please-review PR is ready for (re-)review by a maintainer labels Dec 15, 2025
pothos added a commit to flatcar/scripts that referenced this pull request Dec 16, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
@pothos pothos force-pushed the user-verity-certs-root branch from b2d47ff to ab69fcc Compare December 16, 2025 15:15
@github-actions github-actions bot added please-review PR is ready for (re-)review by a maintainer and removed needs-rebase labels Dec 16, 2025
pothos added a commit to flatcar/scripts that referenced this pull request Dec 17, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
@pothos
Copy link
Copy Markdown
Contributor Author

pothos commented Dec 18, 2025

[ 152.334524][ T5597] request_module fs-erofs succeeded, but still no fs?

This showed up again and it looks like tumbleweed has broken erofs support? Or the way the erofs image is built by repart is not compatible? The test that triggers this is using systemd-repart --make-ddi=sysext. I thought about skipping the test but haven't tried out how to detect this tumbleweed-specific condition. Need to set up a VM…

@pothos pothos force-pushed the user-verity-certs-root branch from ab69fcc to 1e3a528 Compare December 18, 2025 11:56
@pothos
Copy link
Copy Markdown
Contributor Author

pothos commented Dec 18, 2025

This showed up again and it looks like tumbleweed has broken erofs support?

Indeed EROFS is explicitly not supported as @chewi found out on a system he had access to.
I've added a condition to the test.

@pothos pothos force-pushed the user-verity-certs-root branch from 1e3a528 to eda1d41 Compare December 22, 2025 06:55
@daandemeyer
Copy link
Copy Markdown
Collaborator

daandemeyer commented Dec 22, 2025

Given --root= operates entirely below the given root directory, I'm inclined to believe this is the way to go. User certs are rather iffy from a security standpoint anyway, since they're trivially bypassed by just not using our tools to operate on the images.

The verity user certs weren't looked up in the given --root= for
systemd-sysext which made it fail to set up extensions with a strict
image policy.
Look up verity user certs from inside the --root= when we operate on
images in it. The main use case where this matters is when the initrd
sets up the extensions for the final system and thus systemd-sysext
should do the same thing as it would do in the final system.
@pothos pothos force-pushed the user-verity-certs-root branch from eda1d41 to f6acea2 Compare December 23, 2025 07:17
@daandemeyer daandemeyer added good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed and removed please-review PR is ready for (re-)review by a maintainer labels Dec 23, 2025
@yuwata yuwata merged commit fd8a1de into systemd:main Jan 4, 2026
55 of 57 checks passed
@github-actions github-actions bot removed the good-to-merge/waiting-for-ci 👍 PR is good to merge, but CI hasn't passed at time of review. Please merge if you see CI has passed label Jan 4, 2026
pothos added a commit to flatcar/scripts that referenced this pull request Feb 2, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Feb 28, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Feb 28, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Mar 3, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

dissect machine portable Anything to do with systemd-portable and portablectl and portables sysext tests util-lib

Development

Successfully merging this pull request may close these issues.

4 participants