Skip to content

Get rid of debootstrap dependency#1442

Merged
behrmann merged 9 commits intosystemd:mainfrom
daandemeyer:fixes
Apr 17, 2023
Merged

Get rid of debootstrap dependency#1442
behrmann merged 9 commits intosystemd:mainfrom
daandemeyer:fixes

Conversation

@daandemeyer
Copy link
Copy Markdown
Contributor

Let's get rid of our dependency on debootstrap by replicating its
core functionality ourselves. To make sure the necessary tools for
maintainer scripts to run are available in the chroot, we have to
extract the essential debs manually first before installing all the
essential debs.

Additionally, we use the keyring from the host now instead of the one
from the chroot, similar to how we do it for other distros.

This also allows us to get rid of our skeleton tree hack we added for
debootstrap.

@daandemeyer daandemeyer force-pushed the fixes branch 11 times, most recently from 9ae2ad5 to df4cd83 Compare April 16, 2023 09:42
@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 16, 2023

I'm not sure I like the idea of not using debootstrap, it does complicated crap that I'm pretty sure we don't want to have to reinvent and maintain here forever, what's the advantage?

@daandemeyer
Copy link
Copy Markdown
Contributor Author

I'm not sure I like the idea of not using debootstrap, it does complicated crap that I'm pretty sure we don't want to have to reinvent and maintain here forever, what's the advantage?

Much smaller images by default, since we have much larger control over what's installed. Also, we get to use apt's dependency resolution and caching for everything, we don't have to configure both tools to do the right thing, can use more than one mirror which debootstrap doesn't support, it's faster, ... .

The core limitation with debootstrap is that it's built around apt not being available on the host system, which means it has to (usually poorly) re-implement apt. Luckily, we're very much OK with running apt on the host system, and because of that, we're mostly hindered by debootstrap poorly re-implementing apt. We can get much more reliable behavior by just using apt instead of debootstrap.

The only things we have to take care of is setting up merged usr and extracting the essential debs so maintainer scripts can run properly. After that, we delegate everything to apt.

@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 16, 2023

I tried the branch, something is getting lost along the way as there are no apt repos configured anymore

@daandemeyer daandemeyer force-pushed the fixes branch 2 times, most recently from b26e103 to 9b169e5 Compare April 16, 2023 13:13
@daandemeyer
Copy link
Copy Markdown
Contributor Author

daandemeyer commented Apr 16, 2023

I tried the branch, something is getting lost along the way as there are no apt repos configured anymore

So I still really think we shouldn't configure repos by default. We don't do it for any other distro, and to keep things consistent we shouldn't do it for Debian/Ubuntu either.

mkosi is an image builder, and I'm sure we should focus on image based workflows, including image based updates, for which configured package manager repositories in the image aren't needed.

With these changes, we don't even install apt by default anymore in the produced Ubuntu and Debian images (which finally makes them consistent with the other distros where we don't install a package manager either), so putting repositories in the image just doesn't make sense anymore if you ask me.

Or think about other use cases like building an initrd, having apt or configured repositories in such an image doesn't make sense at all, so we shouldn't put them in there by default.

@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 16, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

@daandemeyer
Copy link
Copy Markdown
Contributor Author

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

@daandemeyer
Copy link
Copy Markdown
Contributor Author

Reworked to add the repositories if apt was installed.

@daandemeyer
Copy link
Copy Markdown
Contributor Author

Added some NEWS updates

@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 16, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

I don't think that's possible, as everything is uploaded to unstable, then migrates to testing, and eventually is promoted to stable, same identical binary package, so you cannot have a fixed configuration for one suite

Copy link
Copy Markdown
Member

@keszybz keszybz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bit … involved, but not too horrible.

@keszybz
Copy link
Copy Markdown
Member

keszybz commented Apr 17, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

I don't think that's possible, as everything is uploaded to unstable, then migrates to testing, and eventually is promoted to stable, same identical binary package, so you cannot have a fixed configuration for one suite

This shouldn't be too hard. Have some concept of "distro version" and insert it into a template?
Not being able to install config as a static package seems so 90's.

@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 17, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

I don't think that's possible, as everything is uploaded to unstable, then migrates to testing, and eventually is promoted to stable, same identical binary package, so you cannot have a fixed configuration for one suite

This shouldn't be too hard. Have some concept of "distro version" and insert it into a template? Not being able to install config as a static package seems so 90's.

Nope, the only official way to differentiate between testing and unstable is the apt repositories configuration. Without it, you cannot tell which is which, hence you cannot install the right one.

We now make sure paths are absolute when parsing them so no need to
do so anymore in debian.py
For other distributions, we don't install the repositories at all but
because Debian and Ubuntu do not have an easy way to install the
repositories via a package we give them some leeway and install the
repositories if apt was installed.

We also rework repository handling to accomodate this. When running
apt ourselves on the host, we now use the apt directory in the
workspace as the config directory for apt instead of using etc/apt
in the image.

Additionally, we also use the keyring from the host instead of the
one from the image when running apt.

Finally, we switch to the http mirror for debian security updates
to avoid having to install ca-certificates in the image for apt
to work properly.
Current Debian stable boots fine in qemu with systemd-boot with this
workaround removed, so let's drop it.
Let's continue to stop mucking around with distro defaults and leave
the ignore * preset for users to apply in a postinst script.
@keszybz
Copy link
Copy Markdown
Member

keszybz commented Apr 17, 2023

W: No sandbox user '_apt' on the system, can not drop privileges
— is this something to worry about? Maybe we should provide the user to get rid of the warning even if it isn't strictly necessary.

Let's get rid of our dependency on debootstrap by replicating its
core functionality ourselves. To make sure the necessary tools for
maintainer scripts to run are available in the chroot, we have to
extract the essential debs manually first before installing all the
essential debs.

This also allows us to get rid of our skeleton tree hack we added for
debootstrap.
@daandemeyer
Copy link
Copy Markdown
Contributor Author

W: No sandbox user '_apt' on the system, can not drop privileges
— is this something to worry about? Maybe we should provide the user to get rid of the warning even if it isn't strictly necessary.

The thing is, apt wants this user on the host system, so this would be something to fix in the packages providing apt. Instead, I modified the apt config to use root as the sandbox user. If you care about this, you can run mkosi unprivileged and in that case, root is already sandboxed as we'll be using a user namespace and it should be fine to use as the sandbox user. If you're running mkosi as root, it's full of holes anyway so using root as the sandbox user shouldn't be an issue either.

@keszybz
Copy link
Copy Markdown
Member

keszybz commented Apr 17, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

I don't think that's possible, as everything is uploaded to unstable, then migrates to testing, and eventually is promoted to stable, same identical binary package, so you cannot have a fixed configuration for one suite

This shouldn't be too hard. Have some concept of "distro version" and insert it into a template? Not being able to install config as a static package seems so 90's.

Nope, the only official way to differentiate between testing and unstable is the apt repositories configuration. Without it, you cannot tell which is which, hence you cannot install the right one.

Yeah, this is exactly the problem to fix. Add VERSION_ID=/VERSION_CODENAME= fields to os-release, read the file when installing packages, and substitute {version} in apt.sources as appropriate.

keszybz added a commit to keszybz/mkosi that referenced this pull request Apr 17, 2023
I had this failure when testing systemd#1442. But
it would happen whenever the installed system has no /etc/shadow and we want to
set the password, so it's worth fixing regardless.
Copy link
Copy Markdown
Member

@keszybz keszybz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@bluca
Copy link
Copy Markdown
Member

bluca commented Apr 17, 2023

IIRC the others can initialize themselves, apt cannot. It's not good UX, and it's not how other image builders work so the result is surprising. Maybe add them only if apt is passed to the package list? That way for small images without apt there's no such config file

Can we please make an effort to get a package in Debian that installs the repositories then? e.g. a package debian-release that writes to /etc/apt/sources.list.d/xxx.list what we write in mkosi. We can do it in mkosi for now but we should have a plan to get rid of it.

I don't think that's possible, as everything is uploaded to unstable, then migrates to testing, and eventually is promoted to stable, same identical binary package, so you cannot have a fixed configuration for one suite

This shouldn't be too hard. Have some concept of "distro version" and insert it into a template? Not being able to install config as a static package seems so 90's.

Nope, the only official way to differentiate between testing and unstable is the apt repositories configuration. Without it, you cannot tell which is which, hence you cannot install the right one.

Yeah, this is exactly the problem to fix. Add VERSION_ID=/VERSION_CODENAME= fields to os-release, read the file when installing packages, and substitute {version} in apt.sources as appropriate.

Nope, the VERSION_ID is testing/unstable intentionally https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1021663

@daandemeyer daandemeyer requested a review from behrmann April 17, 2023 10:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants