Skip to content
This repository was archived by the owner on May 30, 2023. It is now read-only.

use sysext in azure OEM images#2506

Closed
krnowak wants to merge 8 commits intomainfrom
krnowak/azure-sysext
Closed

use sysext in azure OEM images#2506
krnowak wants to merge 8 commits intomainfrom
krnowak/azure-sysext

Conversation

@krnowak
Copy link
Copy Markdown
Contributor

@krnowak krnowak commented Mar 10, 2023 •
edited
Loading

app-emulation/wa-linux-agent: Start changing the ebuild to resemble a normal ebuild, without the OEM hacks. This replaces dependency on dev-lang/python-oem and dev-python/distro-oem with dev-lang/python and dev-python/distro, respectively.

dev-lang/python-oem, dev-python/distro-oem: Drop the unnecessary packages.

profiles, coreos-devel/board-packages: Drop things related to the dropped python packages. Maybe we should start pulling in dev-lang/python instead?

CI: http://jenkins.infra.kinvolk.io:8080/job/container/job/packages_all_arches/1472/cldsv

Related scripts PR: flatcar/scripts#683

  • Changelog entries added in the respective changelog/ directory (user-facing change, bug fix, security fix, update)
  • Inspected CI output for image differences: /boot and /usr size, packages, list files for any missing binaries, kernel modules, config files, kernel modules, etc.

@krnowak krnowak added the main label Mar 10, 2023
@krnowak krnowak mentioned this pull request Mar 10, 2023
Closed
2 tasks
pothos
pothos reviewed Mar 13, 2023
View reviewed changes
Comment thread app-emulation/wa-linux-agent/files/0001-flatcar-changes.patch Outdated
pothos
pothos reviewed Mar 13, 2023
View reviewed changes
Comment thread app-emulation/wa-linux-agent/wa-linux-agent-2.6.0.2-r1.ebuild Outdated
@pothos
Copy link
Copy Markdown
Contributor

pothos commented Mar 13, 2023

While we have a workaround in Flatcar to support systemd units symlinks in TARGET.wants/ for enabling a service, this is not upstream behavior and we should rather add a target drop-in unit with Upholds= - this will ensure that the service is started after a manual systemd-sysext refresh.

@pothos
Copy link
Copy Markdown
Contributor

pothos commented Mar 16, 2023

While we have a workaround in Flatcar to support systemd units symlinks in TARGET.wants/ for enabling a service, this is not upstream behavior and we should rather add a target drop-in unit with Upholds= - this will ensure that the service is started after a manual systemd-sysext refresh.

The services should also be put in the sysext. The files coreos-base/oem-azure/files/units/waagent.service and coreos-base/oem-azure/files/base/base.ign should be removed (assuming we use the upstream waagend.service definition, otherwise this should be a downstream patch?) and not installed to disk anymore.

@pothos pothos mentioned this pull request Mar 16, 2023
Draft
2 tasks
@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Mar 16, 2023

There is also an issue with the /usr/share/oem/oem-release file. If coreos-base/oem-azure package is used solely for filling the sysext rootfs, then /usr/share/oem/oem-release will end up being inside sysext. Which means that the sysext won't be even activated.

I see two ways out of it:

  • two packages, one for production image, one for sysext
    • we probably could even try having a generic coreos-base/oem-common (or coreos-base/oem-release) package that installs the oem-release file.
  • one package + special sauce in scripts that creates the oem-release file with specific contents in the production image.

@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Mar 16, 2023

  • two packages, one for production image, one for sysext
    • we probably could even try having a generic coreos-base/oem-common (or coreos-base/oem-release) package that installs the oem-release file.

I'll try this one first.

@pothos
Copy link
Copy Markdown
Contributor

pothos commented Mar 16, 2023

The grub.cfg will also have to be part of either the common package (controlled through use flags as already done in the ec2 oem package)

@pothos pothos mentioned this pull request Mar 17, 2023
Open
9 tasks
@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Mar 17, 2023

While we have a workaround in Flatcar to support systemd units symlinks in TARGET.wants/ for enabling a service, this is not upstream behavior and we should rather add a target drop-in unit with Upholds= - this will ensure that the service is started after a manual systemd-sysext refresh.

I'm not sure I understand, could you expand on it, please? Drop in for multi-user.target that adds Upholds=waagent.service?

@krnowak krnowak force-pushed the krnowak/azure-sysext branch from 706169b to fb317a2 Compare March 17, 2023 13:42
@pothos
Copy link
Copy Markdown
Contributor

pothos commented Mar 17, 2023

I'm not sure I understand, could you expand on it, please? Drop in for multi-user.target that adds Upholds=waagent.service?

Yes, then systemctl daemon-reload done by systemd-sysext is enough to ensure that it runs. The wants-symlinks only work if we also restart all active targets which is a workaround I added for Flatcar but upstream recommends to use Upholds instead. We need to start the service after a systemd-sysext refresh because this temporarily unloads the service unit files, meaning the execution will be stopped.

Here an example:

# cat test.service 
[Unit]
Description=test
[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=echo hello

# cat multi-user.target.d/10-test-service.conf 
[Unit]
Upholds=test.service

# systemctl daemon-reload
# systemctl status test.service

This leads to test.service being started.

@krnowak krnowak force-pushed the krnowak/azure-sysext branch from fb317a2 to f2a6ad8 Compare March 17, 2023 14:40
@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Mar 17, 2023

Cool, thanks! I appreciate it.

@krnowak krnowak force-pushed the krnowak/azure-sysext branch 2 times, most recently from 9db3761 to 4226b6d Compare March 22, 2023 11:51
krnowak added 2 commits March 23, 2023 16:14
@krnowak
coreos-base/coreos-init: Pull in updated azure udev rules
5455d45
@krnowak
unfinished commit
14dacca 
app-emulation/wa-linux-agent: Since the contents of this package will
be now a part of the sysext image, we don't need any special
OEM-specific hacks. We don't need to install the package in
/usr/share/oem directory any more, so update the ebuild to use the
Gentoo python machinery to install files in the usual locations. This
can also use a normal python package, so replace dependencies on
dev-lang/python-oem and dev-python/distro-oem with dev-lang/python and
dev-python/distro, respectively. The waagent.conf file we used to
provide is updated (to disable auto updates, for example, and dropped
obsolete options) and now is a part of the patch, so it is installed
by the python machinery.

dev-lang/python-oem, dev-python/distro-oem: Drop the unnecessary
packages.

profiles, coreos-devel/board-packages: Drop things related to the
dropped python packages.

coreos-base/oem-azure: This package will be used for the sysext image,
instead of for installing files into /usr/share/oem. This means that
we can drop some files or move them elsewhere. The systemd service
file is not needed, because it is installed by the
app-emulation/wa-linux-agent package now. This also means that the
ignition file as lost its purpose. The grub.cfg and oem-release must
be installed in /usr/share/oem, next to the sysext raw image file, so
handling of these files is moved to the newly added
coreos-base/common-oem-files package. `eject` symlink to
`/usr/bin/true` is installed in the newly added manglefs.sh script.

coreos-base/common-oem-files: This is a new package that will generate
and install both grub.cfg and oem-release files into
/usr/share/oem. Each platform can customize the process by providing
their own fragments for each of the two files if necessary.
@krnowak krnowak force-pushed the krnowak/azure-sysext branch from 4226b6d to 14dacca Compare March 23, 2023 15:14
@krnowak krnowak force-pushed the krnowak/azure-sysext branch from 7d08ef9 to 0e68817 Compare April 3, 2023 11:58
@pothos
Copy link
Copy Markdown
Contributor

pothos commented Apr 5, 2023

To migrate existing servers we need to add the list of old OEM files in /etc/ (from the 'base' Ignition config) and the OEM partition to https://github.com/flatcar/bootengine/blob/d3cc0f4b1dce6a5084a8a909810efc30c367020b/dracut/99setup-root/initrd-setup-root-after-ignition#L59

@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Apr 5, 2023

To migrate existing servers we need to add the list of old OEM files in /etc/ (from the 'base' Ignition config) and the OEM partition to https://github.com/flatcar/bootengine/blob/d3cc0f4b1dce6a5084a8a909810efc30c367020b/dracut/99setup-root/initrd-setup-root-after-ignition#L59

Thanks, noted. I have not been yet thinking about the migration path. First wanted to get the fresh deployments to work.

@krnowak
Copy link
Copy Markdown
Contributor Author

krnowak commented Apr 14, 2023

Will update a PR in scripts after Big Merge.

@krnowak krnowak closed this Apr 14, 2023
@krnowak krnowak mentioned this pull request May 31, 2023
Merged
2 tasks
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

1 more reviewer

@pothos pothos pothos left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

main

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@krnowak @pothos