[Python] Build manylinux aarch64 without cross-compilation#40354
[Python] Build manylinux aarch64 without cross-compilation#40354sreenithi wants to merge 54 commits intogrpc:masterfrom
Conversation
templates/tools/dockerfile/grpc_artifact_python_manylinux2014_aarch64/Dockerfile.inja
Show resolved
Hide resolved
asheshvidyut
left a comment
There was a problem hiding this comment.
Nice work with execution time reduction. 👍
sergiitk
left a comment
There was a problem hiding this comment.
Great work, thank you! Feel free to merge once you're confident with the build results / CI status
|
Thanks, all the required tests are actually passing. But I believe sanity was broken at the time the test ran. Have just re-triggered them now |
|
Seems so. I'll keep re-running them. |
|
The time |
|
Out of curiosity, here's list of job sorted by time taken: |
|
vs randomly selected job prior to this PR |
There's our extra hour spent on |
This PR does the following: - updates the base docker images to recent ones for manylinux-x86 and manylinux-x64 - updates Python version to be installed in the armv7 images Related PRs updating other Python docker images: - grpc#40317 - grpc#40354 Closes grpc#40383 COPYBARA_INTEGRATE_REVIEW=grpc#40383 from sreenithi:update_manylinux_docker_base_images dbfe777 PiperOrigin-RevId: 793828964
This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets) - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward. Closes grpc#40354 PiperOrigin-RevId: 798104586
In #40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool). However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg). As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)): ``` TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec] ``` In #40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`. Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156. Closes #40568 COPYBARA_INTEGRATE_REVIEW=#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994 PiperOrigin-RevId: 800598324
This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets) - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward. Closes grpc#40354 PiperOrigin-RevId: 798104586
In grpc#40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool). However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg). As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)): ``` TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec] ``` In grpc#40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`. Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156. Closes grpc#40568 COPYBARA_INTEGRATE_REVIEW=grpc#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994 PiperOrigin-RevId: 800598324
…ts (#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403 Prerequisites merged before this PR: - #40317 - #40354 - #40383 Closes #40289 COPYBARA_INTEGRATE_REVIEW=#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets) - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward. Closes grpc#40354 PiperOrigin-RevId: 798104586
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
…pilation (#40636) Backport of #40354 to v1.75.x. --- This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from #40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets) - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward.
…pilation (grpc#40636) Backport of grpc#40354 to v1.75.x. --- This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward.
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
…ters for distribtests (#40637) Backport of #40289 to v1.75.x. --- This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403 Prerequisites merged before this PR: - #40317 - #40354 - #40383
This PR does the following: - updates the base docker images to recent ones for manylinux-x86 and manylinux-x64 - updates Python version to be installed in the armv7 images Related PRs updating other Python docker images: - grpc#40317 - grpc#40354 Closes grpc#40383 COPYBARA_INTEGRATE_REVIEW=grpc#40383 from sreenithi:update_manylinux_docker_base_images dbfe777 PiperOrigin-RevId: 793828964
This PR does the following: - Updates the base docker image used by the manylinux2014_aarch64 artifact build from a dockcross image that uses cross-compilation to a simple pypa image that doesn't require cross-compilation - Hence switch the manylinux2014_aarch64 build to run in an ARM64 machine without cross compiling. - We already have a '**Distribution Tests Python Linux Arm64**' job that runs on an ARM64 machine to build only `musllinux_aarch64` targets. - Updating the build process so that the manylinux2014_aarch64 target is also now built as part of the '**Distribution Tests Python Linux Arm64**' job instead of the '**Distribution Tests Python Linux**' job. This change cuts the execution time of the '**Distribution Tests Python Linux**' job significantly effectively reducing the total runtime by over 55% as shown below: - Previously the presubmit runs usually take about 75-90 minutes of runtime: [Example run from grpc#40317](https://btx.cloud.google.com/invocations/147e002e-12da-441f-a744-630233725563/targets) - Now the presubmit run-time has reduced to about 30-40 minutes: [Test run from this PR](https://btx.cloud.google.com/invocations/93602c73-228b-48e1-9bcc-059f51479887/targets) Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward. Closes grpc#40354 PiperOrigin-RevId: 798104586
In grpc#40354, we moved all PR/release aarch64 distribtests to the native ARM hardware (`grpc-arm64-small` Kokoro pool). However, we missed the fact there's a separate Python distribtest job that specific to GCP: [`grpc/core/master/linux/grpc_distribtests_gcp_python`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/grpc_distribtests_gcp_python.cfg). As the result, arm64 manulinux jobs build_artifact tasks started timing out ([fail example](https://source.cloud.google.com/results/invocations/2d1bd13c-0e24-4f78-9366-c95ec5c761d4)): ``` TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp310-cp310 [pid=7029, time=7207.9sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp313-cp313 [pid=134112, time=7205.3sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp311-cp311 [pid=134104, time=7205.4sec] TIMEOUT: build_artifact.python_manylinux2014_aarch64_cp312-cp312 [pid=134105, time=7205.5sec] ``` In grpc#40568, we excluded aarch64 tests from `grpc/core/master/linux/grpc_distribtests_gcp_python`. Similar to [`grpc/core/master/linux/arm64/grpc_distribtests_python_arm64`](https://github.com/grpc/grpc/blob/957c045d9aaeaba607cd7883fbb0e2474e3e07f6/tools/internal_ci/linux/arm64/grpc_distribtests_python_arm64.cfg) (master distribtest ARM64) job, we should create a new Kokoro job, `grpc/core/master/linux/arm64/grpc_distribtests_gcp_python_arm64`, and move aarch64 GCP Python distribtests to it. Tracking: b/441764156. Closes grpc#40568 COPYBARA_INTEGRATE_REVIEW=grpc#40568 from sergiitk:fix/build/py/gcp-aarch-filters fd12994 PiperOrigin-RevId: 800598324
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
This PR does the following:
musllinux_aarch64targets.This change cuts the execution time of the 'Distribution Tests Python Linux' job significantly effectively reducing the total runtime by over 55% as shown below:
Given that the presubmit jobs building only a small subset of targets show such time improvement, the release job which currently builds many more aarch64 artifacts will also show significant execution time improvement going forward.