Skip to content

feat(android): add Rust cross-compilation setup for Android environment#2650

Merged
mhsmith merged 11 commits into
pypa:mainfrom
ririv:main
Jan 25, 2026
Merged

feat(android): add Rust cross-compilation setup for Android environment#2650
mhsmith merged 11 commits into
pypa:mainfrom
ririv:main

Conversation

@ririv

@ririv ririv commented Nov 5, 2025

Copy link
Copy Markdown
Contributor

Like platform windows

def setup_rust_cross_compile(
tmp: Path, # noqa: ARG001
python_configuration: PythonConfiguration,
python_libs_base: Path, # noqa: ARG001
env: MutableMapping[str, str],
) -> None:
# Assume that MSVC will be used, because we already know that we are
# cross-compiling. MinGW users can set CARGO_BUILD_TARGET themselves
# and we will respect the existing value.
cargo_target = {
"64": "x86_64-pc-windows-msvc",
"32": "i686-pc-windows-msvc",
"ARM64": "aarch64-pc-windows-msvc",
}.get(python_configuration.arch)
# CARGO_BUILD_TARGET is the variable used by Cargo and setuptools_rust
if env.get("CARGO_BUILD_TARGET"):
if env["CARGO_BUILD_TARGET"] != cargo_target:
log.notice("Not overriding CARGO_BUILD_TARGET as it has already been set")
# No message if it was set to what we were planning to set it to
elif cargo_target:
log.notice(f"Setting CARGO_BUILD_TARGET={cargo_target} for cross-compilation")
env["CARGO_BUILD_TARGET"] = cargo_target
else:
log.warning(
f"Unable to configure Rust cross-compilation for architecture {python_configuration.arch}"
)

Add Rust cross-compilation setup for Android environment, it can fix the cross-compilation problem for the rust project built with setuptools-rust

@ririv ririv requested a review from mhsmith as a code owner November 5, 2025 02:59
@ririv

ririv commented Nov 5, 2025

Copy link
Copy Markdown
Contributor Author

For Rust projects built with Maturin, it currently does not work. Maturin has some cross-compilation issues on Android that still need to be fixed.

For details, please see PyO3/maturin#2810

@joerick

joerick commented Nov 7, 2025

Copy link
Copy Markdown
Contributor

This looks nice and neat! Is there an example of it working in a real project that you've been testing it with?

@ririv

ririv commented Nov 7, 2025

Copy link
Copy Markdown
Contributor Author

I want to test, but I did not know how to run this without the github action,if you know how to do, you can tell me,and I will test it.

This commit is a modification I made based on the actual missing environment configuration built on Android using cbuildwheel.

Of course, I manually configured it externally in a non-isolated environment, and after the configuration, I built the wheel correctly and it can run on Android

@joerick

joerick commented Nov 7, 2025

Copy link
Copy Markdown
Contributor

You can test a project using something like this in a github workflow-

      - name: Build wheels
        uses: ririv/cibuildwheel@main

@ririv ririv force-pushed the main branch 10 times, most recently from 1384753 to 99ee18a Compare November 10, 2025 06:01
…_DIR to link against libpython3.x.so explicitly
@ririv

ririv commented Nov 10, 2025

Copy link
Copy Markdown
Contributor Author

Thanks, I have tested it, and made some new commits. Now, it works.

The PYO3_CROSS_LIB_DIR environment variable should be set here rather than requiring users to specify it manually, for the following reasons:

  • PYO3 has become the de facto standard for Python extension modules built with Rust. Both setuptools-rust and maturin are tools developed by the PYO3 team specifically to serve PYO3.
  • The value set for PYO3_CROSS_LIB_DIR requires the directory path of target_python. This path is already configured within cbuildwheel's android.py but cannot be directly accessed externally. External configuration would necessitate manually constructing the path.
  • PYO3's documentation for Android cross-compilation is insufficiently detailed. If users attempt to set this variable manually, they are likely to become confused and unsure how to proceed.

@ririv

ririv commented Nov 10, 2025

Copy link
Copy Markdown
Contributor Author

https://github.com/ririv/android-wheels/blob/main/.github/workflows/cbuildweel-repair-test.yml
This is the workflow file. It performed some conversions.
For the maturin project, it was converted to a setuptools-rust project. This is because maturin has some bugs in Android cross-compilation. I've submitted a PR to fix it, but the maturin maintainer hasn't responded yet.

Here are some logs:

@joerick

joerick commented Nov 28, 2025

Copy link
Copy Markdown
Contributor

Looks good to me! @mhsmith what do you think?

@mhsmith

mhsmith commented Dec 2, 2025

Copy link
Copy Markdown
Member

Sorry for the delay, I'll look at this as soon as I can.

Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Copilot AI review requested due to automatic review settings December 21, 2025 04:55

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request adds Rust cross-compilation setup for the Android platform, mirroring similar functionality that exists for Windows. The implementation configures environment variables necessary for building Rust-based Python packages (using setuptools-rust and PyO3) when cross-compiling for Android targets.

Key Changes:

  • Added setup_rust_cross_compile() function to configure Cargo build target and linker environment variables
  • Added setup_PYO3_cross_compile() function to set PyO3 cross-compilation library directory
  • Integrated both functions into the Android environment setup process

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
@ririv

ririv commented Dec 22, 2025

Copy link
Copy Markdown
Contributor Author

Thanks for the suggestions. I've updated the PR to address the feedback:

  • Lazy Rust Setup: I implemented the shim/wrapper approach for cargo and rustup as suggested. This ensures zero overhead for non-Rust projects, as the Android target is only installed when a Rust command is actually triggered. I used absolute paths in the shims to avoid recursion.
  • Platform Alignment: Cleaned up the redundant log.notice calls and removed the defensive CC check to follow the android-env.sh contract.
  • New Integration Test: Added test_setuptools_rust to test_android.py. It builds a minimal Rust extension and runs it on the emulator to verify the whole flow.

@ririv

ririv commented Dec 22, 2025

Copy link
Copy Markdown
Contributor Author

Since PyO3/maturin#2825 has been merged into main, we can now use cibuildwheel to test Rust projects built with maturin. However, as a new version hasn't been released yet, we have to build maturin from source for now.

This is the GItHub Action script:

https://github.com/ririv/android-wheels/blob/main/.github/workflows/test-cbuildwheel-repair-maturin-source.yml

This is the logs:


Sorry, there are still some issues with the maturin integration: it's building for Linux instead of Android. This might be a problem with the build module; I will investigate and fix it.

However, we can still use a conversion tool to transform maturin projects into setuptools-rust projects and successfully build Android wheels.

@ririv

ririv commented Dec 26, 2025

Copy link
Copy Markdown
Contributor Author

I have investigated the issue with the wheel generated by maturin. It actually produced the correct wheel, but the tag was incorrect.
This occurred because maturin does not yet support Android platform tags, causing it to mistakenly identify Android as Linux and generate an incorrect tag.
I have submitted a PR to maturin to fix this. Details can be found at PyO3/maturin#2900

This issue is unrelated to the current PR. So, no modification is necessary.
Please review, thanks. @mhsmith

@mhsmith

mhsmith commented Dec 26, 2025

Copy link
Copy Markdown
Member

Thanks; I'm on vacation at the moment, but I'll look at this in the new year.

Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
Comment thread cibuildwheel/platforms/android.py Outdated
@ririv

ririv commented Jan 8, 2026

Copy link
Copy Markdown
Contributor Author

Key Changes:

  1. Resource-based Shim Script:

    • Moved the inline shim script from android.py to a standalone static resource file: cibuildwheel/resources/_rust_shim.py.
    • The shim logic is now consistent with _cross_venv.py. It dynamically detects the real rustup/cargo in $PATH and reads the target triplet from the CIBW_HOST_TRIPLET environment variable at runtime.
  2. Simplified android.py Logic:

    • Removed the conditional check for rustup existence in setup_rust.
    • The setup_rust function now unconditionally installs the shim scripts into the virtual environment's bin directory and sets the necessary environment variables (CARGO_BUILD_TARGET, PYO3_CROSS_LIB_DIR, linkers, etc.).
    • This ensures a lazy setup: rustup target add is only triggered if and when the build process actually calls cargo.
    • Rename: python_configuration -> config
  3. Expanded Test Coverage:

    • Added test_maturin integration test: Since the relevant PR has been merged into maturin (now requiring maturin >= 1.11.0), I've added a new test in test/test_android.py to verify this path.

Ready for re-review. Thanks! @mhsmith

@ririv

ririv commented Jan 8, 2026

Copy link
Copy Markdown
Contributor Author

Here is how the Rust shim script integrates into the build process:

  1. Deployment (The "Trap"):
    During environment setup (setup_android_env), cibuildwheel copies _rust_shim.py into the virtual environment's binary directory (venv/bin/), renaming it to cargo and rustup. Since venv/bin/ is prepended to the system PATH, these shims effectively shadow the system's real Rust binaries.

  2. Interception & Setup (Lazy Execution):
    When the build frontend (e.g., pip or build) invokes cargo, the OS executes our shim script first (via the #!/usr/bin/env python3 shebang). The script reads the target architecture from the CIBW_HOST_TRIPLET environment variable and, if the required Android target is missing, silently executes rustup target add <target> to install it on demand.

  3. Handoff (Transparency):
    Once the environment is ready, the shim locates the real cargo executable in the remaining PATH. It then replaces the current process with the real cargo (using os.execv), passing along all original CLI arguments. This ensures the build continues seamlessly with the correct cross-compilation environment now in place.

@ririv ririv requested a review from mhsmith January 8, 2026 07:53

@mhsmith mhsmith left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, everything looks good now.

@mhsmith mhsmith requested a review from joerick January 17, 2026 18:33

@joerick joerick left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks both for your work on this. Looks good to me.

@mhsmith mhsmith merged commit dc4fc9c into pypa:main Jan 25, 2026
35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants