Skip to content

32-bit x86 zstd static library is built without -fPIC, causing error: relocation R_386_PC32 #8

@robertkirkman

Description

@robertkirkman

Describe the bug

Unfortunately, the Android testbed is not currently buildable for the 32-bit x86 target because the 32-bit x86 build of libzstd.a distributed here:

https://github.com/beeware/cpython-android-source-deps/releases/tag/zstd-1.5.7-1

is causing this error at link-time of CPython when it is built for 32-bit x86 Android:

ld.lld: error: relocation R_386_PC32 cannot be used against symbol 'HUF_writeCTable_wksp'; recompile with -fPIC
>>> defined in /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a(huf_compress.o)
>>> referenced by huf_compress.c
>>>               huf_compress.o:(HUF_optimalTableLog) in archive /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a

The correct way to compile libzstd.a for 32-bit x86 Android is by applying CFLAGS+=" -fPIC" to the build environment. This can be observed in these preexisting projects targeting 32-bit x86 Android:

Termux

(license of build script code in the link: Apache-2.0)

https://github.com/termux/termux-packages/blob/de0dda66f0693e9e3ab8841aea1e15ec8b5da700/scripts/build/toolchain/termux_setup_toolchain_29.sh#L51-L56

Luanti

(license of build script code in the link: CC0-1.0)

https://github.com/luanti-org/luanti_android_deps/blob/3e3a94a9af059e3eabf7b085c7a40f1c5b90c1a7/build.sh#L55

How to reproduce the bug

Build the Dockerfile in the minimum example code section below using this command: docker build --output=. .

Minimum example code

FROM ubuntu:24.04 AS build

# based on
# https://github.com/thunder-coding/containerTube/blob/93062c010527382b0eb5372e9fd0842cb9f9a3f5/python-android-builder/Dockerfile

# Install necessary packages we need to build Python
RUN <<DOCKEREOF
apt-get update && apt-get full-upgrade -y
# Needed for getting the sources and SDK
apt-get install -y git curl unzip
# For actually building Python
apt-get install -y build-essential python-is-python3 autoconf autoconf-archive m4 autotools-dev libffi-dev
# Needed for Android command line tools
apt-get install -y openjdk-21-jdk
# Needed for building zstd in cpython build system
apt-get install -y wget
DOCKEREOF

ADD https://dl.google.com/android/repository/commandlinetools-linux-14742923_latest.zip /root/cmdline-tools.zip
ADD https://github.com/python/cpython/archive/5992238986df094e890a89376970aab6058a0759.tar.gz /root/cpython.tar.gz
ADD https://github.com/beeware/cpython-android-source-deps/archive/refs/tags/zstd-1.5.7-1.tar.gz /root/cpython-android-source-deps.tar.gz

ENV ANDROID_HOME=/root/.android

WORKDIR /root

RUN <<DOCKEREOF
mkdir -p "$ANDROID_HOME/cmdline-tools"
unzip cmdline-tools.zip -d "$ANDROID_HOME/cmdline-tools"
mv "$ANDROID_HOME/cmdline-tools/cmdline-tools" "$ANDROID_HOME/cmdline-tools/latest"
mkdir -p cpython cpython-android-source-deps
tar xf cpython.tar.gz --strip-components=1 -C cpython
tar xf cpython-android-source-deps.tar.gz --strip-components=1 -C cpython-android-source-deps
DOCKEREOF

WORKDIR /root/cpython-android-source-deps

RUN ./build.sh zstd 1.5.7 2 i686-linux-android

WORKDIR /root/cpython

RUN <<DOCKEREOF
cat > /root/git-ls-files.log << 'EOF'
README.md
android-env.sh
android.py
testbed/.gitignore
testbed/.idea/inspectionProfiles/Project_Default.xml
testbed/app/.gitignore
testbed/app/build.gradle.kts
testbed/app/src/androidTest/java/org/python/testbed/PythonSuite.kt
testbed/app/src/main/AndroidManifest.xml
testbed/app/src/main/c/CMakeLists.txt
testbed/app/src/main/c/main_activity.c
testbed/app/src/main/java/org/python/testbed/MainActivity.kt
testbed/app/src/main/res/drawable-xxhdpi/ic_launcher.png
testbed/app/src/main/res/layout/activity_main.xml
testbed/app/src/main/res/values/strings.xml
testbed/build.gradle.kts
testbed/gradle.properties
testbed/gradle/wrapper/gradle-wrapper.properties
testbed/settings.gradle.kts
EOF
git apply -v << 'PATCHEOF'
--- a/Android/android.py
+++ b/Android/android.py
@@ -34,7 +34,7 @@
 TESTBED_DIR = ANDROID_DIR / "testbed"
 CROSS_BUILD_DIR = PYTHON_DIR / "cross-build"
 
-HOSTS = ["aarch64-linux-android", "x86_64-linux-android"]
+HOSTS = ["aarch64-linux-android", "x86_64-linux-android", "arm-linux-androideabi", "i686-linux-android"]
 APP_ID = "org.python.testbed"
 DECODE_ARGS = ("UTF-8", "backslashreplace")
 
@@ -743,7 +743,7 @@ def package(context):
 
         # Include all tracked files from the Android directory.
         for line in run(
-            ["git", "ls-files"],
+            ["cat", "/root/git-ls-files.log"],
             cwd=ANDROID_DIR, capture_output=True, text=True, log=False,
         ).stdout.splitlines():
             src = ANDROID_DIR / line
--- a/Android/testbed/app/build.gradle.kts
+++ b/Android/testbed/app/build.gradle.kts
@@ -16,6 +16,8 @@ val inSourceTree = (
 val KNOWN_ABIS = mapOf(
     "aarch64-linux-android" to "arm64-v8a",
     "x86_64-linux-android" to "x86_64",
+    "arm-linux-androideabi" to "armeabi-v7a",
+    "i686-linux-android" to "x86",
 )
 
 // Discover prefixes.
--- a/Lib/sysconfig/__init__.py
+++ b/Lib/sysconfig/__init__.py
@@ -697,12 +697,16 @@ def get_platform():
         # When Python is running on 32-bit ARM Android on a 64-bit ARM kernel,
         # 'os.uname().machine' is 'armv8l'. Such devices run the same userspace
         # code as 'armv7l' devices.
+        # During the build process of the Android testbed when targeting 32-bit ARM,
+        # '_PYTHON_HOST_PLATFORM' is 'arm-linux-androideabi', so 'machine' becomes
+        # 'arm'.
         machine = {
             "x86_64": "x86_64",
             "i686": "x86",
             "aarch64": "arm64_v8a",
             "armv7l": "armeabi_v7a",
             "armv8l": "armeabi_v7a",
+            "arm": "armeabi_v7a",
         }[machine]
     elif osname == "linux":
         # At least on Linux/Intel, 'machine' is the processor --
--- a/Lib/test/test_sysconfig.py
+++ b/Lib/test/test_sysconfig.py
@@ -376,6 +376,7 @@ def test_get_platform(self):
             'aarch64': 'arm64_v8a',
             'armv7l': 'armeabi_v7a',
             'armv8l': 'armeabi_v7a',
+            'arm': 'armeabi_v7a',
         }.items():
             with self.subTest(machine):
                 self._set_uname(('Linux', 'localhost', '3.18.91+',
@@ -585,6 +586,7 @@ def test_android_ext_suffix(self):
             "aarch64": "aarch64-linux-android",
             "armv7l": "arm-linux-androideabi",
             "armv8l": "arm-linux-androideabi",
+            "arm": "arm-linux-androideabi",
         }[machine]
         self.assertEndsWith(suffix, f"-{expected_triplet}.so")
 
PATCHEOF
DOCKEREOF

WORKDIR /root/cpython/Android

RUN <<DOCKEREOF
./android.py configure-build
./android.py make-build
DOCKEREOF

RUN <<DOCKEREOF
./android.py configure-host arm-linux-androideabi
./android.py make-host arm-linux-androideabi
./android.py package arm-linux-androideabi
mv ../cross-build/arm-linux-androideabi/dist/python-*-arm-linux-androideabi.tar.gz \
    /root/python-arm-linux-androideabi.tar.gz
DOCKEREOF

WORKDIR /root/cpython

RUN <<DOCKEREOF
git apply -v << 'PATCHEOF'
--- a/Android/android.py
+++ b/Android/android.py
@@ -214,6 +214,8 @@ def unpack_deps(host, prefix_dir):
         download(f"{deps_url}/{name_ver}/{filename}")
         shutil.unpack_archive(filename)
         os.remove(filename)
+    shutil.copyfile("/root/cpython-android-source-deps/zstd/build/1.5.7/i686-linux-android/prefix/lib/libzstd.a",
+        "/root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a")
 
 
 def download(url, target_dir="."):
PATCHEOF
DOCKEREOF

WORKDIR /root/cpython/Android

RUN <<DOCKEREOF
./android.py configure-host i686-linux-android
./android.py make-host i686-linux-android
./android.py package i686-linux-android
mv ../cross-build/i686-linux-android/dist/python-*-i686-linux-android.tar.gz \
    /root/python-i686-linux-android.tar.gz
DOCKEREOF

FROM scratch
COPY --from=build /root/python-arm-linux-androideabi.tar.gz /
COPY --from=build /root/python-i686-linux-android.tar.gz /

Screenshots

No response

Environment details

  • Operating system and version: 32-bit x86 Android 5
  • Python version: Python 3.15 development commit 5992238986df094e890a89376970aab6058a0759

Logs

ld.lld: error: relocation R_386_PC32 cannot be used against symbol 'HUF_writeCTable_wksp'; recompile with -fPIC
>>> defined in /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a(huf_compress.o)
>>> referenced by huf_compress.c
>>>               huf_compress.o:(HUF_optimalTableLog) in archive /root/cpython/cross-build/i686-linux-android/prefix/lib/libzstd.a

Additional context

Related issue:

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions