images/arch, images/ubuntu: Add p11-kit-client.so#1634
Merged
debarshiray merged 5 commits intocontainers:mainfrom May 1, 2025
Merged
Conversation
a4a1208 to
620d55a
Compare
Member
Author
|
The However, I can't reproduce it locally with: @Jmennius @travier Any idea? I wonder if it's a problem with the infrastructure. Maybe a stale buggy |
Collaborator
That's weird... doesn't seem to be a transient error (i rerun the job), but at the same time it doesn't reproduce locally (on the same base image, according to hashes). |
The system tests can be very I/O intensive, because many of them copy OCI images from the test suite's image cache directory to its local container/storage store, create containers, and then delete everything to run the next test with a clean slate. This makes them slow. The tests for toolbox(1) invocations forwarded to the host, which includes the help tests, are particularly slow because they never skip the I/O. This makes them a good target for optimizations. The help tests for toolbox(1) invocations forwarded to the host use the same default Toolbx container to invoke toolbox(1) from without changing its state. Therefore, a lot of disk I/O can be avoided by creating the default container only once for all those tests. This can reduce the time needed to run the help tests from almost 7 minutes to a few seconds. containers#1635
Member
Author
Yes, I guess it's some CDN or mirror something. Thanks for checking! |
The system tests can be very I/O intensive, because many of them copy OCI images from the test suite's image cache directory to its local container/storage store, create containers, and then delete everything to run the next test with a clean slate. This makes them slow. The runtime environment tests, which includes the group and user tests, are particularly slow because they don't skip the I/O even when testing error handling. This makes them a good target for optimizations. The group and user tests check the group and user configuration in different containers without changing their state. Therefore, a lot of disk I/O can be avoided by creating these containers only once for all the tests. This can reduce the time needed to run the group and user tests from almost 22 minutes to almost 5 minutes. containers#1635
Use a strings.Builder [1] instance instead of raw string literals for consistency. Fallout from 772b66b [1] https://pkg.go.dev/strings#Builder containers#1636
A subsequent commit will use this to give Toolbx containers access to the certificates from certificate authorities on the host. This commit was kept separate from the changes to toolbox(1) to ensure that the ubuntu-toolbox images are ready before that happens. Unfortunately, Ubuntu 16.04 Xenial Xerus has p11-kit 0.23.2 [1], while p11-kit-client.so was introduced in p11-kit 0.23.10 [2]. Therefore, this feature will be disabled for Ubuntu 16.04. This reverts parts of commit 1e93086 to re-enable building the ubuntu-toolbox:18.04 image. [1] https://code.launchpad.net/~git-ubuntu-import/ubuntu/+source/p11-kit/+git/p11-kit/+ref/ubuntu/xenial-updates [2] p11-kit commit 0684cd7b7f815b41 p11-glue/p11-kit@0684cd7b7f815b41 p11-glue/p11-kit#15 containers#626
620d55a to
aa85077
Compare
The libp11-kit package was added to the arch-toolbox image to ensure the presence of p11-kit-client.so. Currently, the package is already pulled in by various dependencies, like the gnutls and p11-kit packages. Therefore, it doesn't increase the size of the base image, but serves as a safeguard against any inadvertent changes. A subsequent commit will use this to give Toolbx containers access to the certificates from certificate authorities on the host. This commit was kept separate from the changes to toolbox(1) to ensure that the arch-toolbox image is ready before that happens. containers#626
Member
Author
|
cc @Foxboron |
This was referenced May 1, 2025
debarshiray
added a commit
to debarshiray/toolbox
that referenced
this pull request
May 10, 2025
This uses the same approach taken by Flatpak [1] to ensure that the certificates from certificate authorities (or CAs) that are available inside a Toolbx container are kept synchronized with the host operating system. Any program that uses PKCS containers#11 to access CA certificates should see the same ones both inside the container and on the host. During every 'enter' and 'run' command, toolbox(1) ensures that an instance of 'p11-kit server' is running on the host listening on a local file system socket that's accessible to both the container and the host. If an instance is already running, then a second one is not created. The location of the socket is injected into the container through the P11_KIT_SERVER_ADDRESS environment variable. The Toolbx container's entry point configures it to use the p11-kit-client.so PKCS containers#11 module instead of the usual p11-kit-trust.so module. This talks to the 'p11-kit server' instance running on the host over the socket instead of reading the CA certificates that are present inside the container. However, unlike Flatpak, this doesn't use D-Bus to set up the communication between the container and the host, because when invoked as 'sudo toolbox ...' there's no user or session D-Bus instance available for the root user. This set-up is skipped if 'p11-kit server' can't be run on the host, or if the /etc/pkcs11/modules directory for configuring PKCS containers#11 modules or p11-kit-client.so are missing inside the container. None of these are considered hard dependencies to accommodate size-constrained OSes like Fedora CoreOS that might not have 'p11-kit server', and existing Toolbx containers and old images that might not have p11-kit-client.so. Verions of lsof from 4.88 to older than 4.94.0 always select the file descriptor field [2], and these are present in versions of Ubuntu from 16.04 to 22.04. This needs to be factored in as long as a Ubuntu 22.04 host is used to run the CI. One option is to always select the file descriptor field even though it's not really needed for the tests. The ubuntu-toolbox:22.04 has been failing to rebuild [3]: STEP 6/10: RUN sed -Ei '/apt-get (update|upgrade)/s/^/#/' /usr/local/sbin/unminimize && apt-get update && yes | /usr/local/sbin/unminimize && DEBIAN_FRONTEND=noninteractive apt-get -y install ubuntu-minimal ubuntu-standard libnss-myhostname flatpak-xdg-utils $(cat extra-packages | xargs) && rm -rd /var/lib/apt/lists/* ... ... (Reading database ... 100% (Reading database ... 4387 files and directories currently installed.) Preparing to unpack .../libc-bin_2.35-0ubuntu3.9_arm64.deb ... Unpacking libc-bin (2.35-0ubuntu3.9) over (2.35-0ubuntu3.9) ... Setting up libc-bin (2.35-0ubuntu3.9) ... qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) dpkg: error processing package libc-bin (--configure): installed libc-bin package post-installation script subprocess returned error exit status 139 Errors were encountered while processing: libc-bin E: Sub-process /usr/bin/dpkg returned an error code (1) ... It's probably some network problem because it doesn't reliably occur everywhere. Until this gets sorted out, Toolbx containers created from the ubuntu-toolbox:22.04 won't have access to the CA certificates from the host. The fedora-toolbox and the UBI-based toolbox images haven't yet been updated to contain p11-kit-client.so. Until that happens, containers created from them won't have access to the CA certificates from the host. [1] Flatpak commit 66b2ff40f7caf3a7 flatpak/flatpak@66b2ff40f7caf3a7 flatpak/flatpak#1757 p11-glue/p11-kit#68 [2] lsof commit 811dc78cc6404cb3 lsof-org/lsof@811dc78cc6404cb3 lsof-org/lsof#125 lsof-org/lsof#103 [3] containers#1634 containers#1643 containers#626
debarshiray
added a commit
to debarshiray/toolbox
that referenced
this pull request
May 10, 2025
This uses the same approach taken by Flatpak [1] to ensure that the certificates from certificate authorities (or CAs) that are available inside a Toolbx container are kept synchronized with the host operating system. Any program that uses PKCS containers#11 to access CA certificates should see the same ones both inside the container and on the host. During every 'enter' and 'run' command, toolbox(1) ensures that an instance of 'p11-kit server' is running on the host listening on a local file system socket that's accessible to both the container and the host. If an instance is already running, then a second one is not created. The location of the socket is injected into the container through the P11_KIT_SERVER_ADDRESS environment variable. The Toolbx container's entry point configures it to use the p11-kit-client.so PKCS containers#11 module instead of the usual p11-kit-trust.so module. This talks to the 'p11-kit server' instance running on the host over the socket instead of reading the CA certificates that are present inside the container. However, unlike Flatpak, this doesn't use D-Bus to set up the communication between the container and the host, because when invoked as 'sudo toolbox ...' there's no user or session D-Bus instance available for the root user. This set-up is skipped if 'p11-kit server' can't be run on the host, or if the /etc/pkcs11/modules directory for configuring PKCS containers#11 modules or p11-kit-client.so are missing inside the container. None of these are considered hard dependencies to accommodate size-constrained OSes like Fedora CoreOS that might not have 'p11-kit server', and existing Toolbx containers and old images that might not have p11-kit-client.so. Verions of lsof from 4.88 to older than 4.94.0 always select the file descriptor field [2], and these are present in versions of Ubuntu from 16.04 to 22.04. This needs to be factored in as long as a Ubuntu 22.04 host is used to run the CI. One option is to always select the file descriptor field even though it's not really needed for the tests. The ubuntu-toolbox:22.04 has been failing to rebuild [3]: STEP 6/10: RUN sed -Ei '/apt-get (update|upgrade)/s/^/#/' /usr/local/sbin/unminimize && apt-get update && yes | /usr/local/sbin/unminimize && DEBIAN_FRONTEND=noninteractive apt-get -y install ubuntu-minimal ubuntu-standard libnss-myhostname flatpak-xdg-utils $(cat extra-packages | xargs) && rm -rd /var/lib/apt/lists/* ... ... (Reading database ... 100% (Reading database ... 4387 files and directories currently installed.) Preparing to unpack .../libc-bin_2.35-0ubuntu3.9_arm64.deb ... Unpacking libc-bin (2.35-0ubuntu3.9) over (2.35-0ubuntu3.9) ... Setting up libc-bin (2.35-0ubuntu3.9) ... qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) qemu: uncaught target signal 11 (Segmentation fault) - core dumped Segmentation fault (core dumped) dpkg: error processing package libc-bin (--configure): installed libc-bin package post-installation script subprocess returned error exit status 139 Errors were encountered while processing: libc-bin E: Sub-process /usr/bin/dpkg returned an error code (1) ... It's probably some network problem because it doesn't reliably occur everywhere. Until this gets sorted out, Toolbx containers created from the ubuntu-toolbox:22.04 won't have access to the CA certificates from the host. The fedora-toolbox and the UBI-based toolbox images haven't yet been updated to contain p11-kit-client.so. Until that happens, containers created from them won't have access to the CA certificates from the host. [1] Flatpak commit 66b2ff40f7caf3a7 flatpak/flatpak@66b2ff40f7caf3a7 flatpak/flatpak#1757 p11-glue/p11-kit#68 [2] lsof commit 811dc78cc6404cb3 lsof-org/lsof@811dc78cc6404cb3 lsof-org/lsof#125 lsof-org/lsof#103 [3] containers#1634 containers#1643 containers#626
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A subsequent commit will use this to give Toolbx containers access to
the certificates from certificate authorities on the host. This commit
was kept separate from the changes to
toolbox(1)to ensure that thearch-toolboxandubuntu-toolboximages are ready before thathappens.
#626