bump(main/python): 3.13 by thunder-coding · Pull Request #27739 · termux/termux-packages

thunder-coding · 2025-12-25T09:16:22Z

Mostly ready for review, waiting for CI to turn green and some testing to ensure nothing is broken. Every change is in python-fullrebuild branch, so it is recommend to review that branch. Branch python313 as well as python313-dep{1,2,...} branches are derived from the python-fullrebuild branch. The list of packages that needed to be rebuilt was generated using apt-file search lib/python3.12. Those packages were than later replaced with their parent package names (in case of subpackages), and then some scripts were used to intelligently sort the packages into buckets for rebuild. The list of packages was then saved to output/rebuildlist.txt

Script used to generate rebuild list. This script was specifically designed to take into account large packages that needed more disk space for rebuild as well as packages that depended on python and also were dependencies of other python packages:

#!/usr/bin/bash
set -eou pipefail

rm -f output/rebuildlist-*.txt

declare -a packages=()
while IFS= read -r package; do
	packages+=("$package")
done < output/rebuildlist.txt

orig_packages=("${packages[@]}")
orig_grep_str=""
for pkg in "${orig_packages[@]}"; do
	orig_grep_str+="|$pkg"
done
orig_grep_str="${orig_grep_str:1}"

current_index=1

_generate_rebuild_list() {
	declare -a new_packages=()
	declare -a curr_packages=()
	local grep_str=""
	local -a args=("$@")
	for ((i = 0; i < $#; i++)); do
		pkg="${args[$i]}"
		grep_str+="|$pkg"
	done
	grep_str="${grep_str:1}"
	# echo "DEBUG: Grep string is ($grep_str)"
	for package in "${packages[@]}"; do
		local extra_dirs=()
		if [[ ! -f "packages/$package/build.sh" ]]; then
			extra_dirs+=("packages")
		elif [[ ! -f "x11-packages/$package/build.sh" ]]; then
			extra_dirs+=("x11-packages")
		elif [[ ! -f "root-packages/$package/build.sh" ]]; then
			extra_dirs+=("root-packages")
		fi
		for pkg in "$@"; do
			if [[ "$pkg" == "$package" ]]; then
				curr_packages+=("$package")
				continue 2
			fi
		done
		if ./scripts/buildorder.py ./*packages/"$package" "${extra_dirs[@]}" | grep -qE "($grep_str)$"; then
			curr_packages+=("$package")
			continue
		fi
		new_packages+=("$package")
	done

	local rebuild_list=()
	for pkg in "${curr_packages[@]}"; do
		local extra_dirs=()
		if [[ ! -f "packages/$pkg/build.sh" ]]; then
			extra_dirs+=("packages")
		elif [[ ! -f "x11-packages/$pkg/build.sh" ]]; then
			extra_dirs+=("x11-packages")
		elif [[ ! -f "root-packages/$pkg/build.sh" ]]; then
			extra_dirs+=("root-packages")
		fi
		for rebuild_pkg in $(./scripts/buildorder.py ./*packages/"$pkg" "${extra_dirs[@]}" | grep -oE "($orig_grep_str)$"); do
			rebuild_list+=("$rebuild_pkg")
		done
		rebuild_list+=("$pkg")
	done
	# Remove duplicates from rebuild_list
	rebuild_list=($(echo "${rebuild_list[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

	echo "DEBUG: Dealt with $(( ${#packages[@]} - ${#new_packages[@]} )) packages, ${#new_packages[@]} remain. This step

Script to generate python313-dep* branches:

#!/usr/bin/bash
set -eou pipefail

rm -f output/rebuildlist-*.txt

declare -a packages=()
while IFS= read -r package; do
	packages+=("$package")
done < output/rebuildlist.txt

orig_packages=("${packages[@]}")
orig_grep_str=""
for pkg in "${orig_packages[@]}"; do
	orig_grep_str+="|$pkg"
done
orig_grep_str="${orig_grep_str:1}"

current_index=1

_generate_rebuild_list() {
	declare -a new_packages=()
	declare -a curr_packages=()
	local grep_str=""
	local -a args=("$@")
	for ((i = 0; i < $#; i++)); do
		pkg="${args[$i]}"
		grep_str+="|$pkg"
	done
	grep_str="${grep_str:1}"
	# echo "DEBUG: Grep string is ($grep_str)"
	for package in "${packages[@]}"; do
		local extra_dirs=()
		if [[ ! -f "packages/$package/build.sh" ]]; then
			extra_dirs+=("packages")
		elif [[ ! -f "x11-packages/$package/build.sh" ]]; then
			extra_dirs+=("x11-packages")
		elif [[ ! -f "root-packages/$package/build.sh" ]]; then
			extra_dirs+=("root-packages")
		fi
		for pkg in "$@"; do
			if [[ "$pkg" == "$package" ]]; then
				curr_packages+=("$package")
				continue 2
			fi
		done
		if ./scripts/buildorder.py ./*packages/"$package" "${extra_dirs[@]}" | grep -qE "($grep_str)$"; then
			curr_packages+=("$package")
			continue
		fi
		new_packages+=("$package")
	done

	local rebuild_list=()
	for pkg in "${curr_packages[@]}"; do
		local extra_dirs=()
		if [[ ! -f "packages/$pkg/build.sh" ]]; then
			extra_dirs+=("packages")
		elif [[ ! -f "x11-packages/$pkg/build.sh" ]]; then
			extra_dirs+=("x11-packages")
		elif [[ ! -f "root-packages/$pkg/build.sh" ]]; then
			extra_dirs+=("root-packages")
		fi
		for rebuild_pkg in $(./scripts/buildorder.py ./*packages/"$pkg" "${extra_dirs[@]}" | grep -oE "($orig_grep_str)$"); do
			rebuild_list+=("$rebuild_pkg")
		done
		rebuild_list+=("$pkg")
	done
	# Remove duplicates from rebuild_list
	rebuild_list=($(echo "${rebuild_list[@]}" | tr ' ' '\n' | sort -u | tr '\n' ' '))

	echo "DEBUG: Dealt with $(( ${#packages[@]} - ${#new_packages[@]} )) packages, ${#new_packages[@]} remain. This step rebuilds $((${#rebuild_list[@]} - ${#curr_packages[@]})) extra packages."

	for pkg in "${curr_packages[@]}"; do
		echo "$pkg"
	done > output/rebuildlist-$current_index.txt
	for pkg in "${rebuild_list[@]}"; do
		for curr_pkg in "${curr_packages[@]}"; do
			if [[ "$pkg" == "$curr_pkg" ]]; then
				continue 2
			fi
		done
		echo "$pkg"
	done > output/rebuildlist-$current_index-extra.txt

	packages=("${new_packages[@]}")
	((current_index++))
}
_generate_rebuild_list libllvm flang python-llvmlite python-libsass xapp asciidoc
_generate_rebuild_list python-torch pyqt5
_generate_rebuild_list pygobject pycairo opencv dbus-python python-lxml
_generate_rebuild_list libxml2
_generate_rebuild_list python-numpy python-greenlet python-msgpack python-brotli python-pillow python-scipy
_generate_rebuild_list boost 2ping borgbackup fetchmail speechd vapoursynth
_generate_rebuild_list python-cryptography python-skia-pathops python-pycryptodomex libunbound libsearpc
_generate_rebuild_list "${packages[@]}"

Only the python313 branch needs to be generated full from scratch before running the above script

Warning

As of writing this. the list below (list of todos along with rebuild list) is of the local changes I have, and may not be in sync with the branches and PRs on github.

The following is the significant part that needs to be done first:

Move away from Ubuntu's python when building cross python. We need to bootstrap same version of build-python first. Thankfully I had done this a few days ago for an unrelated project, so I have the build scripts with me for an older version that works, I just need to port it for Python 3.13
Fix ctypes builtin package not detecting packages for cross-python. ctypes.utils.find_package() isn't working. (Tracked down to python/cpython@1f8b24e)
Get all things building (tested locally for aarch64. Waiting for CI to turn green)
Get pip packages to build
Full test suite locally on Termux + termux-docker, like I do for nodejs

Caution

This is a huge BREAKING CHANGE. Seems like upstream changed a lot of stuff which is going to break a lot of stuff for users. Major breaking changes which I found will affect us most:

Note

Since nobody seems to have started this work since a long time, I am also thinking of taking up maintainership of python. Will try to keep this up to date with full test coverage.

`sys.platform` now reports `"android"`

Earlier versions of uptil 3.12 used to have sys.platform to "linux"

import sys

# Prints "android" on Python >= 3.13
# and    "linux"   on Python <= 3.12
print(sys.platform)

This will very likely break scripts that assume Android to just be Linux earlier. Might need new changes in packages downstream. This is going to be definitely a lot of fun to deal with. Although shouldn't be that much pain considering Android is now a supported platform

Rebuild Step 1 #27768