Skip to content

systemd-sysext: introduce a global config#38250

Merged
keszybz merged 3 commits intosystemd:mainfrom
esposem:sysext
Sep 17, 2025
Merged

systemd-sysext: introduce a global config#38250
keszybz merged 3 commits intosystemd:mainfrom
esposem:sysext

Conversation

@esposem
Copy link
Copy Markdown
Contributor

@esposem esposem commented Jul 17, 2025

This PR implements what is proposed in #37992.

Having a global config file that supports the same cmdline options for sysext/confext allows the user to customize the behavior of systemd-sysext.service unit too, without the need of hacking the service manually.

The global config will live in CONF_PATHS_STRV()/systemd/{sysext/confext}.conf and it will be overridden by cmdline, so it is possible to customize a run if systemd-sysext is executed manually.

For now support --mutable= (Mutable) and --image-policy= (ImagePolicy).

@github-actions github-actions bot added documentation util-lib sysext please-review PR is ready for (re-)review by a maintainer labels Jul 17, 2025
@esposem
Copy link
Copy Markdown
Contributor Author

esposem commented Jul 17, 2025

@keszybz please take a look

@bluca
Copy link
Copy Markdown
Member

bluca commented Jul 17, 2025

As mentioned in the issue and in the chat, I don't think these options should be first-class citizens like this. They are not to be used in production, they are debugging/development tools, as they make the entire setup a completely pointless exercise.

As mentioned, these utilities really do not seem like the right solution for your requirements, and there are more promising options available to shave some time off the installation step.

@keszybz
Copy link
Copy Markdown
Member

keszybz commented Jul 17, 2025

As mentioned in the issue and in the chat, I don't think these options should be first-class citizens like this. They are not to be used in production, they are debugging/development tools, as they make the entire setup a completely pointless exercise.

They already are first-class citizens, settable through options and thoroughly documented in the man page. And using those options makes a lot of sense for certain scenarios. They might be used in your preferred deployment method, but that hardly makes them pointless. Please, let's not turn this into a philosophical discussion about the best image deployment methods. We have the options and the policies exactly because there are different scenarios, tradeoffs, and opinions.

@keszybz
Copy link
Copy Markdown
Member

keszybz commented Jul 17, 2025

Signed-off-by: ...

We don't use sob in the system repo.

@keszybz keszybz added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed please-review PR is ready for (re-)review by a maintainer labels Jul 17, 2025
@github-actions github-actions bot added build-system meson please-review PR is ready for (re-)review by a maintainer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jul 18, 2025
@keszybz
Copy link
Copy Markdown
Member

keszybz commented Jul 18, 2025

Looks all reasonable. I think the patches are fine, but since @bluca had doubts about the overall usefulness of this, I won't set the green flag yet. Other folks should weigh in.

@bluca
Copy link
Copy Markdown
Member

bluca commented Jul 18, 2025

As mentioned in the issue and in the chat, I don't think these options should be first-class citizens like this. They are not to be used in production, they are debugging/development tools, as they make the entire setup a completely pointless exercise.

They already are first-class citizens, settable through options and thoroughly documented in the man page. And using those options makes a lot of sense for certain scenarios. They might be used in your preferred deployment method, but that hardly makes them pointless. Please, let's not turn this into a philosophical discussion about the best image deployment methods. We have the options and the policies exactly because there are different scenarios, tradeoffs, and opinions.

Discussing use cases in PRs is quite common, and mostly expected, to understand the scope and purpose of a set of proposed changes. It's not about personal preferences or philosophical discussions so much, but about making sure the changes make sense for the use case, and the right tool is used for the right job.

The use case is not really defined here very well - it was however discussed in chat, and when drilling down it adds up to using a complex filesystem setup in order to save some number of seconds when a VM is installed. Discussing this use case, and whether it makes sense to enhance a tool that has a different purpose to support it, should be fair game in a PR.

If someone wants to deploy writable images, that's absolutely fine of course. The question is not whether that's the best kind of image, but whether using this tool and a complex setup is really the best way to do that, or whether there are alternatives that do not require code changes to be maintained, and achieve the same or close-enough purpose.

@keszybz
Copy link
Copy Markdown
Member

keszybz commented Jul 18, 2025

Sure, obviously we should sometimes discuss the general approach and use cases in pull requests. Nevertheless, in this particular case, we have a situation where the PR in itself is straightforward, it adds a config file for a service following the standard pattern. The config file doesn't add any new functionality, it just fills in a missing bit in a style that is provided for many other services. Those two settings are clearly "policy", and it reasonable to allow them to be set through a vendor config file. The only point of contention is if/when/whether mutable sysexts should be used to construct a file system. I feel that this is not the right place to discuss this, because even if we have the general consensus that sysexts should not be used for the particular purpose that motivated the PR, the PR in itself provides generic functionality that is reasonable and desirable in any case.

Copy link
Copy Markdown
Member

@keszybz keszybz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, I see that I actually forgot to submit the review.

@yuwata yuwata added reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks and removed please-review PR is ready for (re-)review by a maintainer labels Jul 18, 2025
@github-actions github-actions bot added please-review PR is ready for (re-)review by a maintainer and removed reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks labels Jul 21, 2025
@esposem
Copy link
Copy Markdown
Contributor Author

esposem commented Jul 21, 2025

Applied all changes except [Extension]. I think having a common config for sysext and confext makes it complex if sysext has to support an option (say Mutable=true) and confext for whatever reason doesn't.

@keszybz keszybz removed the please-review PR is ready for (re-)review by a maintainer label Jul 23, 2025
@keszybz
Copy link
Copy Markdown
Member

keszybz commented Jul 23, 2025

I set the green label, because the code seems straightforward. But let's do it after the release so that we can have more time to figure out if this is the right approach.

@yuwata yuwata added the reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks label Jul 23, 2025
@github-actions github-actions bot removed the reviewed/needs-rework 🔨 PR has been reviewed and needs another round of reworks label Jul 24, 2025
esposem added 3 commits July 29, 2025 05:34
Introduce systemd/{sysext/confext}.conf and systemd/{sysext/confext}.conf.d to provide an
alternative way of setting the cmdline options in systemd-sysext.

The config file has to have a [Sysext] or [Confext] option respectively,
which will be overridden by the cmdline.

As an example of supported config, add Mutable= option.
Add sysext.conf, which similar to other configs like coredump, will be
searched in:
/{etc run usr/lib}/systemd/{sysext/confext}.conf
but also
/{etc run usr/lib}/systemd/{sysext/confext}.conf.d/*

This config is an alternative to command line options, especially useful
if we want to extend the service units without modifying them.
Just as Mutable=, support ImagePolicy in systemd/{sysext/confext}.conf and
dropins in systemd/{sysext.confext}.conf.d/* configs.
@esposem
Copy link
Copy Markdown
Contributor Author

esposem commented Jul 29, 2025

Rebased and all changes are now applied

@esposem esposem requested a review from yuwata July 31, 2025 13:22
@jcpunk
Copy link
Copy Markdown
Contributor

jcpunk commented Sep 5, 2025

Adding another workflow for use of this.

My edge devices want to generate ssh keys on boot. I don't care what the keys are, upon power off and power on I want the system back into my defined state. With a confext making /etc read-only ssh never comes up. I'm setting ephemeral by hand in the tooling.

@bluca
Copy link
Copy Markdown
Member

bluca commented Sep 5, 2025

Add a fixed ssh config that points to /run/ for the keys

@jcpunk
Copy link
Copy Markdown
Contributor

jcpunk commented Sep 5, 2025

Add a fixed ssh config that points to /run/ for the keys

That also entails updating the selinux/apparmor policy and a few other tweaks. Whereas an ephemeral volume lets me leave the system defaults alone and focus on just my requirements.

@bluca
Copy link
Copy Markdown
Member

bluca commented Sep 5, 2025

Sure, you can do that already if you want to, overlayfs wasn't invented here, it's been in use for almost 20 years now

@jcpunk
Copy link
Copy Markdown
Contributor

jcpunk commented Sep 8, 2025

Should noexec be added to the option list? /etc/X11/xinit/Xsession has an exec bit and can no longer be run when confext sets /etc to noexec.

@keszybz keszybz merged commit 5510a98 into systemd:main Sep 17, 2025
50 of 54 checks passed
@keszybz
Copy link
Copy Markdown
Member

keszybz commented Sep 17, 2025

Should noexec be added to the option list? /etc/X11/xinit/Xsession has an exec bit and can no longer be run when confext sets /etc to noexec.

That's a good question, though not directly related to this PR, since this is just adding a new mechanism to set the policy. If you think something should be changed, open a new issue or PR.

@pothos
Copy link
Copy Markdown
Contributor

pothos commented Oct 12, 2025

It's good that this landed because as soon as we would want to mount the extensions for the final system already from the initrd, then systemd-sysext in the initrd should have a way to use the correct options. The drop-in workarounds with a modified ExecStart line are ignored in this scenario and we need a real config file.

pothos added a commit to flatcar/scripts that referenced this pull request Dec 16, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Dec 17, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Dec 18, 2025
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Feb 2, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Feb 28, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Feb 28, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
pothos added a commit to flatcar/scripts that referenced this pull request Mar 3, 2026
For A/B-updated /etc contents we used a custom overlay mount that
provides the default files through a lowerdir loaded from /usr. Since
then we upstreamed mutable systemd-confext support and now we can switch
to it.
This pulls in flatcar/init#138 and
flatcar/bootengine#115 together with backported
systemd patches that have opened or merged upstream PRs to fix --root=
issues and add a refresh skip check to prevent boot disruptions due to
the multiple daemon reloads and - more important - the missing atomic
remount that would mean /etc is gone for a few milliseconds during boot.
The skip logic works best with verity hashes and thus the default
confext must be a verity extension image.
User-provided confext don't work well yet unless they use verity due to
the missing atomic remount and reliance on the skipping logic. We also
need to look into stacking order and other mutabiliy settings.

The backported systemd patches relate to the following upstream PRs:

systemd/systemd#39843 for
vpick-Don-t-use-openat-directly-but-resolve-symlinks
discover-image-Follow-symlinks-in-a-given-root
sysext-Use-correct-image-name-for-extension-release
test-Add-tests-for-handling-symlinks-with-systemd-sy
Note that the patch in the PR relies on
0859fe3f32774f1e0c787974cc252ff922a1b868 but the backport patch not.

systemd/systemd#39980 for
sysext-Create-mutable-directory-with-the-right-mode
sysext-Skip-refresh-if-no-changes-are-found

systemd/systemd#39991 for
sysext-Get-verity-user-certs-from-given-root

systemd/systemd#40063 for
sysext-Fix-config-file-support-with-root
which relies on systemd/systemd#38250 for
man-sysext.conf-add-systemd-sysext-config-files
sysext-introduce-global-config-file
sysext-support-ImagePolicy-global-config-option

Signed-off-by: Kai Lueke <kailuke@microsoft.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Development

Successfully merging this pull request may close these issues.

6 participants