Skip to content

python 3.13 rebuild: part 5#28181

Closed
thunder-coding wants to merge 47 commits intomasterfrom
python313-dep5
Closed

python 3.13 rebuild: part 5#28181
thunder-coding wants to merge 47 commits intomasterfrom
python313-dep5

Conversation

@thunder-coding
Copy link
Member

No description provided.

@thunder-coding thunder-coding marked this pull request as draft January 26, 2026 06:21
@thunder-coding thunder-coding mentioned this pull request Jan 26, 2026
5 tasks
@thunder-coding thunder-coding force-pushed the python313-dep5 branch 3 times, most recently from ce5d5dd to 288c5a3 Compare January 31, 2026 07:57
@thunder-coding thunder-coding force-pushed the python313-dep5 branch 2 times, most recently from 296fa6f to 7a0daad Compare March 1, 2026 21:51
Earlier 54G -> 25G
Now 53G -> 16G

(10G excluded for docker image)
Does this work? Yes
Is this cursed? Not as cursed as using bash for build system
This should allow for more flexible configuration of docker containers
by allowing devs to pass on their own flags to docker

Also ~/clean.sh will now not remove /home/builder/.termux-build to
account for cases where /home/builder/.termux-build is a volume mounted
to the docker image, where it is not possible to remove the directory
We now are always using docker for builds, so can't do this
Do not clutter what we do in "Gather build summary" step
Also make sure we use this optimization in package_updates.yml
~/.termux-build on host

Should be helpful for local builds for using IDEs and host tools for
development
In commits, %ci:free-space will force freeing space in commits
In workflow dispatch, a new checkbox should be available
Instead of only supporting one of the flags, we now support passing
multiple flags at the same time for more convenience. The command line
argument parser will exit as soon as it detects an argument/flag it
doesn't handle to preserve maximum compatibility with existing commands
**BREAKING CHANGES**

This now requires AppArmor to be installed and running with docker for
limiting the capabilities that `CAP_SYS_ADMIN` provides to containers.

Host kernel must support fuse. The host /dev/fuse device is passed onto
the container.

**DETAILED DESCRIPTION**

./scripts/run-docker.sh first starts with relaxed profile, and then
after changing the uid and gid of the builder user and group, drops to
restricted profile. Each container get's it's own profile so that if
./scripts/run-docker.sh is run parallel with multiple containers, there
is no race condition for the when we are changing the builder uid/gid,
where the other container will run with higher privileges than needed.

For ensuring least privileges, only mount and umount2 syscalls have been
permitted in seccomp profile. Additionally rules for allowing clone,
clone3 and similar syscalls when certain contain conditions are met and
only when CAP_SYS_ADMIN is not set have been removed as we aren't
allowing these syscalls when CAP_SYS_ADMIN is set.

The AppArmor profile is based on Docker's default AppArmor profile. The
profile was extracted using nerdctl (which is an alternate CLI interface
to Docker CLI). The profile can be extracted using
`nerdctl apparmor inspect`. There are two AppArmor profiles we have
setup, one restricted and relaxed. Currently there is little difference
between relaxed and restricted profile. The only difference is that
relaxed profile allows any kind of mount syscall, while the restricted
profile only allows mount syscalls only for fuse.fuse-overlayfs

Regarding security of passing /dev/fuse to containers, the Linux kernel
documentation specifies that it should be fine to pass this to
namespaces. The CAP_SYSTEM_ADMIN is needed only for the mount syscall to
work. Due to some historic reasons this needs this dangerous capability.
Although the syscall is needed we are only allowing mount and umount
syscalls to happen inside the container, so seccomp profile and apparmor
profile should be doing the damage control.

Linux kernel documentation for fuse-passthrough: https://docs.kernel.org/6.16/filesystems/fuse-passthrough.html
Upstream Linux kernel commit for fuse documentation: torvalds/linux@18ee43c

Even without apparmor, things should be fine as we aren't fiddling
around much with apparmor for security reasons, just currently limiting
where the fuse filesystem can be used.

In future, AppArmor profiles can also be used for further hardening of
docker image
@licy183 and me for Seccomp profile

Me for apparmor profile

Feel free to add yourself if you believe that you can deal with this
nicely.

Mostly for others looking for maintenance of the apparmor profile. Just
grab the docker's default config using nerdctl apparmor inspect, diff
with the current config and figure it out

For the seccomp profile, just diff with the exact commit of moby/moby's
seccomp profile and store the updated JSON
AppArmor isn't configured by default on distributions other than Ubuntu,
so don't mandate it. AppArmor proper configuration and setup is a huge
pain especially if you aren't familiar with it and containers in
general.

Even a lot of the maintainers aren't already familiar with it and using
it already so let's just keep it optional and do not use it if not
detected on host.
only respect TERMUX_DOCKER_USE_SUDO for running docker commands

Even if docker is setup in rootless mode, apparmor needs to be run with
sudo. This was the cause of CI failure and me trying to bang my head on
why things weren't working on CI. Thanks, a "sudo" is all it takes. I
wish it worked in real life as well on people
We assume in a lot of places that all files in termux-packages are
trusted. So make sure that the container isn't able to modify anything
in this environment. Also do not allow any changes to ~/lib/ where we
are storing Android NDK and SDK.
thunder-coding and others added 27 commits March 2, 2026 04:38
patch for configure is no longer needed

patch for multiprocessing.c no longer required due to
python/cpython@fa1d675

Remove 0006-do-not-use-xattr.patch since we can just add a configure
argument and it works just as well

We no longer need to manually add libraries for libpython as it is now
added by default when building for android. See upstream commit:
python/cpython@7f5e3f0

No longer required patch for Lib/aifc.py after
python/cpython@036da3b

mailcap module has been removed in Python 3.13.
https://docs.python.org/3/whatsnew/3.13.html#whatsnew313-pep594

add post install message when site-packages/ is detected for python3.12

Also mark as unsafe for on-device building. The plan is to move to building
python with host-python, this isn't possible on device. I will be
looking to get some fixes made in future to get on-device builds as
well, but it is not the priority at the moment
Causes the built libpython3.so to not include symbols which we need in
the final binary
maybe the issue with the build system is fixed now?
We need build python for cross compilation as Ubuntu's version of python
is different. Also crossenv recommends building own python revisions
instead of distribution compiled ones

added lld to setup-ubuntu as it's needed by clang driver when using
-fuse-ld=lld. Not sure how we went this far without having to deal with
this during host builds

Also we need libpython3.so for cross-compilation of some of the pip
packages. So build with --enable-shared
Relevant upstream commit python/cpython@1f8b24e

Also apply this same fix in host build of python as cross-python uses
it's own cutils instead of what we have built for target in our cross
builds.
[[TODO: add more descriptive commit message]]
ac_cv_func_getlogin_r=no is needed to ensure that it doesn't try to use
getlogin_r on Android during on-device builds.

Although I have enabled on-device builds, I don't see any point in
keeping this as the same version of python needs to be available for
building the same version of python. Pretty much defeats the purpose
unless you are rebuilding with some changes for the same minor version
%ci:free-disk
@thunder-coding
Copy link
Member Author

Merged in #27739

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant