Skip to content

[Python] Build manylinux aarch64 without cross-compilation#40354

Closed
sreenithi wants to merge 54 commits intogrpc:masterfrom
sreenithi:update_manylinux_aarch64_build
Closed

[Python] Build manylinux aarch64 without cross-compilation#40354
sreenithi wants to merge 54 commits intogrpc:masterfrom
sreenithi:update_manylinux_aarch64_build

Conversation

@sreenithi
Copy link
Contributor

@sreenithi sreenithi commented Jul 29, 2025

This PR does the following:

  • Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation
  • Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling.
    • We already have a 'Distribution Tests Python Linux Arm64' job that runs on an ARM64 machine to build only musllinux_aarch64 targets.
    • Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the 'Distribution Tests Python Linux Arm64' job instead of the 'Distribution Tests Python Linux' job.

This change cuts the execution time of the 'Distribution Tests Python Linux' job significantly effectively reducing the total runtime by over 55% as shown below:

Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward.

@sreenithi sreenithi changed the title [Python] Update manylinux aarch64 build [Python] Update manylinux aarch64 build without cross-compilation Jul 29, 2025
@sreenithi sreenithi added the release notes: no Indicates if PR should not be in release notes label Jul 29, 2025
@sreenithi sreenithi self-assigned this Jul 29, 2025
@sreenithi sreenithi marked this pull request as ready for review July 31, 2025 04:48
@sreenithi sreenithi requested a review from sergiitk as a code owner July 31, 2025 04:48
@sreenithi sreenithi requested a review from asheshvidyut July 31, 2025 04:48
Copy link
Member

@asheshvidyut asheshvidyut left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work with execution time reduction. 👍

Copy link
Member

@sergiitk sergiitk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work, thank you! Feel free to merge once you're confident with the build results / CI status

@sreenithi
Copy link
Contributor Author

Thanks, all the required tests are actually passing. But I believe sanity was broken at the time the test ran. Have just re-triggered them now

@sergiitk
Copy link
Member

Seems so. I'll keep re-running them.

@sergiitk
Copy link
Member

The time Distribution Tests Python Linux takes is down from 1.5 hours to 0.5 hours on average, and without any impact on Distribution Tests Python Linux Arm64 (around 10 minutes like before). This is great!

@sergiitk
Copy link
Member

Out of curiosity, here's list of job sorted by time taken:

25.3 sec    build_package.python_package_any
25.5 sec    build_package.python_package
63.5 sec    distribtest.python_linux_x86_bullseye
67.8 sec    distribtest.python_linux_x64_bullseye
346.8 sec   build_artifact.python_linux_extra_armv7_cp313-cp313
351.6 sec   build_artifact.python_linux_extra_armv7_cp39-cp39
450.5 sec   build_artifact.python_musllinux_1_2_x86_cp313-cp313
463.2 sec   build_artifact.python_manylinux2014_x86_cp313-cp313
470.6 sec   build_artifact.python_musllinux_1_2_x86_cp39-cp39
471.0 sec   build_artifact.python_manylinux2014_x86_cp39-cp39
472.9 sec   build_artifact.python_musllinux_1_2_x64_cp313-cp313
481.9 sec   build_artifact.python_musllinux_1_2_x64_cp39-cp39
503.9 sec   build_artifact.python_manylinux2014_x64_cp313-cp313
511.9 sec   build_artifact.python_manylinux2014_x64_cp39-cp39
904.2 sec   distribtest.python_dev_linux_x86_bullseye
945.7 sec   distribtest.python_dev_linux_x64_bullseye
1025.8 sec  distribtest.python_dev_linux_x64_alpine3.18

@sergiitk
Copy link
Member

vs randomly selected job prior to this PR

19.1 sec    build_package.python_package
65.9 sec    distribtest.python_linux_x86_bullseye
69.2 sec    distribtest.python_linux_x64_bullseye
471.1 sec   distribtest.python_linux_aarch64_python39_buster
544.0 sec   build_artifact.python_linux_extra_armv7_cp313-cp313
553.8 sec   build_artifact.python_linux_extra_armv7_cp39-cp39
654.6 sec   build_artifact.python_musllinux_1_2_x64_cp313-cp313
658.2 sec   build_artifact.python_musllinux_1_2_x86_cp313-cp313
663.4 sec   build_artifact.python_musllinux_1_2_x86_cp39-cp39
667.3 sec   build_artifact.python_musllinux_1_2_x64_cp39-cp39
676.1 sec   build_artifact.python_manylinux2014_x86_cp313-cp313
680.7 sec   build_artifact.python_manylinux2014_x86_cp39-cp39
686.0 sec   build_artifact.python_manylinux2014_x64_cp313-cp313
694.0 sec   build_artifact.python_manylinux2014_x64_cp39-cp39
1008.4 sec  distribtest.python_dev_linux_x86_bullseye
1041.5 sec  distribtest.python_dev_linux_x64_bullseye
1180.4 sec  distribtest.python_dev_linux_x64_alpine3.18
3969.8 sec  build_artifact.python_manylinux2014_aarch64_cp313-cp313
3980.7 sec  build_artifact.python_manylinux2014_aarch64_cp39-cp39

@sergiitk
Copy link
Member

In [14]: humanize.precisedelta(dt.timedelta(seconds=3969.8))
Out[14]: '1 hour, 6 minutes and 9.80 seconds'

There's our extra hour spent on build_artifact.python_manylinux2014_aarch64_cp313-cp313

paulosjca pushed a commit to paulosjca/grpc that referenced this pull request Aug 23, 2025
This PR does the following:
- updates the base docker images to recent ones for manylinux-x86 and manylinux-x64
- updates Python version to be installed in the armv7 images

Related PRs updating other Python docker images:
- grpc#40317
- grpc#40354

Closes grpc#40383

COPYBARA_INTEGRATE_REVIEW=grpc#40383 from sreenithi:update_manylinux_docker_base_images dbfe777
PiperOrigin-RevId: 793828964
paulosjca pushed a commit to paulosjca/grpc that referenced this pull request Aug 23, 2025
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling.
  - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets.
  - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets)
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant  execution time improvement going forward.

Closes grpc#40354

PiperOrigin-RevId: 798104586
copybara-service bot pushed a commit that referenced this pull request Aug 28, 2025
In #40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool).

However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg).

As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)):
```
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec]
```

In #40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`.

Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156.

Closes #40568

COPYBARA_INTEGRATE_REVIEW=#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994
PiperOrigin-RevId: 800598324
anniefrchz pushed a commit to anniefrchz/grpc that referenced this pull request Sep 3, 2025
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling.
  - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets.
  - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets)
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant  execution time improvement going forward.

Closes grpc#40354

PiperOrigin-RevId: 798104586
anniefrchz pushed a commit to anniefrchz/grpc that referenced this pull request Sep 3, 2025
In grpc#40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool).

However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg).

As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)):
```
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec]
```

In grpc#40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`.

Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156.

Closes grpc#40568

COPYBARA_INTEGRATE_REVIEW=grpc#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994
PiperOrigin-RevId: 800598324
copybara-service bot pushed a commit that referenced this pull request Sep 4, 2025
…ts (#40289)

This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403

Prerequisites merged before this PR:
- #40317
- #40354
- #40383

Closes #40289

COPYBARA_INTEGRATE_REVIEW=#40289 from sreenithi:python_314_test_support 3b8bf96
PiperOrigin-RevId: 802967999
sreenithi added a commit to sreenithi/grpc that referenced this pull request Sep 8, 2025
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling.
  - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets.
  - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets)
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant  execution time improvement going forward.

Closes grpc#40354

PiperOrigin-RevId: 798104586
sreenithi added a commit to sreenithi/grpc that referenced this pull request Sep 8, 2025
…ts (grpc#40289)

This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403

Prerequisites merged before this PR:
- grpc#40317
- grpc#40354
- grpc#40383

Closes grpc#40289

COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96
PiperOrigin-RevId: 802967999
sergiitk pushed a commit that referenced this pull request Sep 8, 2025
…pilation (#40636)

Backport of #40354 to v1.75.x.
---
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64
artifact build from a dockcross image that uses cross-compilation to a
simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64
machine without cross compiling.
- We already have a '**Distribution Tests Python Linux Arm64**' job that
runs on an ARM64 machine to build only `musllinux_aarch64` targets.
- Updating the build process so that the manylinux2014_aarch64 target is
also now built as part of the '**Distribution Tests Python Linux
Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python
Linux**' job significantly effectively reducing the total runtime by
over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of
runtime: [Example run from
#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets)
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test
run from this
PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets
show such time improvement, the release job which currently builds many
more aarch64 artifacts will also show significant execution time
improvement going forward.
sreenithi added a commit to sreenithi/grpc that referenced this pull request Sep 9, 2025
…pilation (grpc#40636)

Backport of grpc#40354 to v1.75.x.
---
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64
artifact build from a dockcross image that uses cross-compilation to a
simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64
machine without cross compiling.
- We already have a '**Distribution Tests Python Linux Arm64**' job that
runs on an ARM64 machine to build only `musllinux_aarch64` targets.
- Updating the build process so that the manylinux2014_aarch64 target is
also now built as part of the '**Distribution Tests Python Linux
Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python
Linux**' job significantly effectively reducing the total runtime by
over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of
runtime: [Example run from
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test
run from this
PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets
show such time improvement, the release job which currently builds many
more aarch64 artifacts will also show significant execution time
improvement going forward.
sreenithi added a commit to sreenithi/grpc that referenced this pull request Sep 9, 2025
…ts (grpc#40289)

This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403

Prerequisites merged before this PR:
- grpc#40317
- grpc#40354
- grpc#40383

Closes grpc#40289

COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96
PiperOrigin-RevId: 802967999
sergiitk pushed a commit that referenced this pull request Sep 9, 2025
…ters for distribtests (#40637)

Backport of #40289 to v1.75.x.
---
This PR adds Python 3.14 support for Distribtests by updating relevant
docker images and distribtest build files for Linux, MacOS and Windows
environments. However, artifact generation in distribtests will be
enabled in the next PR #40403

Prerequisites merged before this PR:
- #40317
- #40354
- #40383
asheshvidyut pushed a commit to asheshvidyut/grpc that referenced this pull request Sep 12, 2025
This PR does the following:
- updates the base docker images to recent ones for manylinux-x86 and manylinux-x64
- updates Python version to be installed in the armv7 images

Related PRs updating other Python docker images:
- grpc#40317
- grpc#40354

Closes grpc#40383

COPYBARA_INTEGRATE_REVIEW=grpc#40383 from sreenithi:update_manylinux_docker_base_images dbfe777
PiperOrigin-RevId: 793828964
asheshvidyut pushed a commit to asheshvidyut/grpc that referenced this pull request Sep 12, 2025
This PR does the following:
- Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation
- Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling.
  - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets.
  - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job.

This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below:
- Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets)
- Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets)

Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant  execution time improvement going forward.

Closes grpc#40354

PiperOrigin-RevId: 798104586
asheshvidyut pushed a commit to asheshvidyut/grpc that referenced this pull request Sep 12, 2025
In grpc#40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool).

However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg).

As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)):
```
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec]
TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec]
```

In grpc#40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`.

Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156.

Closes grpc#40568

COPYBARA_INTEGRATE_REVIEW=grpc#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994
PiperOrigin-RevId: 800598324
asheshvidyut pushed a commit to asheshvidyut/grpc that referenced this pull request Sep 12, 2025
…ts (grpc#40289)

This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403

Prerequisites merged before this PR:
- grpc#40317
- grpc#40354
- grpc#40383

Closes grpc#40289

COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96
PiperOrigin-RevId: 802967999
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants