Skip to content

python 3.13 rebuild: part 3#27977

Closed
thunder-coding wants to merge 57 commits intomasterfrom
python313-dep3
Closed

python 3.13 rebuild: part 3#27977
thunder-coding wants to merge 57 commits intomasterfrom
python313-dep3

Conversation

@thunder-coding
Copy link
Member

@thunder-coding thunder-coding commented Jan 10, 2026

No description provided.

@thunder-coding thunder-coding mentioned this pull request Jan 10, 2026
5 tasks
@thunder-coding thunder-coding force-pushed the python313-dep3 branch 5 times, most recently from 18a52d9 to ae6c557 Compare January 31, 2026 07:53
@thunder-coding thunder-coding force-pushed the python313-dep3 branch 2 times, most recently from 79f3fea to 8cb71d4 Compare March 1, 2026 21:50
Earlier 54G -> 25G
Now 53G -> 16G

(10G excluded for docker image)
Does this work? Yes
Is this cursed? Not as cursed as using bash for build system
This should allow for more flexible configuration of docker containers
by allowing devs to pass on their own flags to docker

Also ~/clean.sh will now not remove /home/builder/.termux-build to
account for cases where /home/builder/.termux-build is a volume mounted
to the docker image, where it is not possible to remove the directory
We now are always using docker for builds, so can't do this
Do not clutter what we do in "Gather build summary" step
Also make sure we use this optimization in package_updates.yml
~/.termux-build on host

Should be helpful for local builds for using IDEs and host tools for
development
In commits, %ci:free-space will force freeing space in commits
In workflow dispatch, a new checkbox should be available
Instead of only supporting one of the flags, we now support passing
multiple flags at the same time for more convenience. The command line
argument parser will exit as soon as it detects an argument/flag it
doesn't handle to preserve maximum compatibility with existing commands
**BREAKING CHANGES**

This now requires AppArmor to be installed and running with docker for
limiting the capabilities that `CAP_SYS_ADMIN` provides to containers.

Host kernel must support fuse. The host /dev/fuse device is passed onto
the container.

**DETAILED DESCRIPTION**

./scripts/run-docker.sh first starts with relaxed profile, and then
after changing the uid and gid of the builder user and group, drops to
restricted profile. Each container get's it's own profile so that if
./scripts/run-docker.sh is run parallel with multiple containers, there
is no race condition for the when we are changing the builder uid/gid,
where the other container will run with higher privileges than needed.

For ensuring least privileges, only mount and umount2 syscalls have been
permitted in seccomp profile. Additionally rules for allowing clone,
clone3 and similar syscalls when certain contain conditions are met and
only when CAP_SYS_ADMIN is not set have been removed as we aren't
allowing these syscalls when CAP_SYS_ADMIN is set.

The AppArmor profile is based on Docker's default AppArmor profile. The
profile was extracted using nerdctl (which is an alternate CLI interface
to Docker CLI). The profile can be extracted using
`nerdctl apparmor inspect`. There are two AppArmor profiles we have
setup, one restricted and relaxed. Currently there is little difference
between relaxed and restricted profile. The only difference is that
relaxed profile allows any kind of mount syscall, while the restricted
profile only allows mount syscalls only for fuse.fuse-overlayfs

Regarding security of passing /dev/fuse to containers, the Linux kernel
documentation specifies that it should be fine to pass this to
namespaces. The CAP_SYSTEM_ADMIN is needed only for the mount syscall to
work. Due to some historic reasons this needs this dangerous capability.
Although the syscall is needed we are only allowing mount and umount
syscalls to happen inside the container, so seccomp profile and apparmor
profile should be doing the damage control.

Linux kernel documentation for fuse-passthrough: https://docs.kernel.org/6.16/filesystems/fuse-passthrough.html
Upstream Linux kernel commit for fuse documentation: torvalds/linux@18ee43c

Even without apparmor, things should be fine as we aren't fiddling
around much with apparmor for security reasons, just currently limiting
where the fuse filesystem can be used.

In future, AppArmor profiles can also be used for further hardening of
docker image
@licy183 and me for Seccomp profile

Me for apparmor profile

Feel free to add yourself if you believe that you can deal with this
nicely.

Mostly for others looking for maintenance of the apparmor profile. Just
grab the docker's default config using nerdctl apparmor inspect, diff
with the current config and figure it out

For the seccomp profile, just diff with the exact commit of moby/moby's
seccomp profile and store the updated JSON
AppArmor isn't configured by default on distributions other than Ubuntu,
so don't mandate it. AppArmor proper configuration and setup is a huge
pain especially if you aren't familiar with it and containers in
general.

Even a lot of the maintainers aren't already familiar with it and using
it already so let's just keep it optional and do not use it if not
detected on host.
only respect TERMUX_DOCKER_USE_SUDO for running docker commands

Even if docker is setup in rootless mode, apparmor needs to be run with
sudo. This was the cause of CI failure and me trying to bang my head on
why things weren't working on CI. Thanks, a "sudo" is all it takes. I
wish it worked in real life as well on people
We assume in a lot of places that all files in termux-packages are
trusted. So make sure that the container isn't able to modify anything
in this environment. Also do not allow any changes to ~/lib/ where we
are storing Android NDK and SDK.
thunder-coding and others added 27 commits March 2, 2026 05:01
This is a slightly modified version of
robotpy/crossenv@3ef761b

We only need the simple shebang, so only use that always.
Also update setuptools to 80.9.0 while we are at full rebuilds with
python 3.13
Relevant upstream commit python/cpython@1f8b24e

Also apply this same fix in host build of python as cross-python uses
it's own cutils instead of what we have built for target in our cross
builds.
…roid api version

`python -m build` builds android wheels as android_$TERMUX_ARCH.whl but
pip during installation expects it to be
android_$TERMUX_API_LEVEL_$TERMUX_ARCH.whl so we need make sure that it
is installable without renaming the wheel
[[TODO: add more descriptive commit message]]
ac_cv_func_getlogin_r=no is needed to ensure that it doesn't try to use
getlogin_r on Android during on-device builds.

Although I have enabled on-device builds, I don't see any point in
keeping this as the same version of python needs to be available for
building the same version of python. Pretty much defeats the purpose
unless you are rebuilding with some changes for the same minor version
Needed to fix hardcoded python version in find_package for Boost.Python.
Needed to fix builds for ledger
Also use ninja for building pyarrow for faster builds than unix
makefiles. Also remove logic for unsetting PYTHONPATH, as it is needed
for Cython to be used.
The libxml2-python subpackage needs to be rebuilt
%ci:free-disk
@thunder-coding
Copy link
Member Author

Merged in #27739

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant