[Python][Support 3.14] Enable Python 3.14 interpreters for distribtests#40289
[Python][Support 3.14] Enable Python 3.14 interpreters for distribtests#40289sreenithi wants to merge 76 commits intogrpc:masterfrom
Conversation
This reverts commit 3ac433b.
templates/tools/dockerfile/grpc_artifact_python_manylinux2014_aarch64/Dockerfile.inja
Show resolved
Hide resolved
tools/bazelify_tests/test/BUILD
Outdated
| name = "artifact_python_linux_x64_manylinux2014_cp313", | ||
| build_script = "build_artifact_python_linux_x64_cp313.sh", | ||
| name = "artifact_python_linux_x64_manylinux2014_cp314", | ||
| build_script = "build_artifact_python_linux_x64_cp314.sh", |
There was a problem hiding this comment.
Will this work with artifact_targets.py change reverted (as we plan to move it to another PR)?
There was a problem hiding this comment.
logically, this should work given that the docker images will have Python 3.14 installed and we set the PYTHON and PIP paths accordingly in build_artifact_python_linux_x64_cp314.sh.
However, this is another variant of the distribtests that checks building artifacts as part of the bazelify tests. Since we decided to enable Python 3.14 artifacts only in PR #40403, I think it makes sense to do this change in that PR only, so have reverted this change too.
tools/internal_ci/helper_scripts/install_python_interpreters.ps1
Outdated
Show resolved
Hide resolved
|
|
||
| # this an aarch64 image | ||
| FROM arm64v8/python:3.13-alpine | ||
| FROM arm64v8/python:3.14.0b4-alpine |
There was a problem hiding this comment.
I see we're using the rc1 in other places, and b4 in others. Looks like this image (arm64v8/python) does have an rc1: https://hub.docker.com/layers/arm64v8/python/3.14.0rc2-alpine/images/sha256-11abbd4209bbadc70256c373654e47974bdafbd36ef9d5d68e30a4f2d9038384
There was a problem hiding this comment.
right, good catch. I believe I made this change even before rc1 was released. So this image and armv7 image was not updated to use the rc1 version.
I've updated the armv7 version in this PR, however I found that this particular tools/dockerfile/distribtest/python_alpine_aarch64 image is used in the distribtest to test the built artifacts.
Given that we want to enable 3.14 artifacts only in the next PR, this image should also be updated only in that one only. So reverting back this file's change to use Python 3.13 in this PR.
This reverts commit 3327ac3.
|
Verified that all the related tests are passing, and the distribtests are not building Python 3.14 artifacts too. So going ahead and submitting it. |
… tests (#40403) This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. ## Distribtests Required pre-requisite changes to enable 3.14 artifacts are covered in #40289 . ## Bazel tests Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in #40602 ## Basic tests The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ### 1) No running event loop for AsyncIO when run outside an async function ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in #39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR #40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ### 2) Pickling error from the `multiprocessing` library ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method. Closes #40403 PiperOrigin-RevId: 804290760
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
…ters for distribtests (#40637) Backport of #40289 to v1.75.x. --- This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403 Prerequisites merged before this PR: - #40317 - #40354 - #40383
…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999
… tests (grpc#40403) This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. ## Distribtests Required pre-requisite changes to enable 3.14 artifacts are covered in grpc#40289 . ## Bazel tests Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in grpc#40602 ## Basic tests The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ### 1) No running event loop for AsyncIO when run outside an async function ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in grpc#39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR grpc#40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ### 2) Pickling error from the `multiprocessing` library ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method. Closes grpc#40403 PiperOrigin-RevId: 804290760
… tests (grpc#40403) This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. Required pre-requisite changes to enable 3.14 artifacts are covered in grpc#40289 . Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in grpc#40602 The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in grpc#39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR grpc#40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method. Closes grpc#40403 PiperOrigin-RevId: 804290760
#40726) Backport of #40403 to v1.75.x. --- This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. ## Distribtests Required pre-requisite changes to enable 3.14 artifacts are covered in #40289 . ## Bazel tests Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in #40602 ## Basic tests The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ### 1) No running event loop for AsyncIO when run outside an async function ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in #39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR #40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ### 2) Pickling error from the `multiprocessing` library ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method.
This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403
Prerequisites merged before this PR: