[Python][Support 3.14] Enable 3.14 in Python Basic, Bazel and Distrib tests by sreenithi · Pull Request #40403 · grpc/grpc

sreenithi · 2025-08-06T09:10:07Z

This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions.

Distribtests

Required pre-requisite changes to enable 3.14 artifacts are covered in #40289 .

Bazel tests

Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in #40602

Basic tests

The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR:

1) No running event loop for AsyncIO when run outside an async function

Traceback (most recent call last):
  File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop
RuntimeError: no running event loop

This was caught by the tests_aio.unit.outside_init_test.TestOutsideInit and tests_aio.unit.init_test.TestInit tests, and was also previously reported in #39507 with the root cause.

Following some investigation, the fix is being worked on by @sergiitk in PR #40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above.

2) Pickling error from the `multiprocessing` library

_pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types
when serializing dict item '_target'
when serializing multiprocessing.context.Process state
when serializing multiprocessing.context.Process object

This was caught by the tests.unit._dynamic_stubs_test.DynamicStubTest which runs test cases in a subprocess using the multiprocessing library.
Error root cause:

The default start method of multiprocessing in linux has changed to forkserver instead of fork from Python 3.14.
forkserver has a few extra restrictions for picklability as compared to fork (Ref: Python Docs)
All the test case functions in the DynamicStubTest that are provided as target to the multiprocessing.Process use decorators. This causes problems when pickling them.

Hence to resolve this, we manually set the 'start method' of multiprocessing to use the fork start method.

asheshvidyut · 2025-08-11T12:26:32Z

src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi

+    not be set as the default event loop for the main thread.
+    """
+    try:
+        return asyncio.get_running_loop()


This method returns a running loop, but in the new method introduced, we are not returning a running loop.

Question - Do we need to make the loop running before returning?

Good point, but we're just retaining the previous behaviour here. Even previously, we returned a running loop only if it was already running, and otherwise just return an existing (not running) event loop returned by asyncio.get_event_loop_policy().get_event_loop() or a new loop returned by asyncio.get_event_loop_policy().new_event_loop().

We're using the same behaviour here, and in addition also setting this newly created loop as the current event loop for the thread using asyncio.set_event_loop(loop).

And following that, we bind to the event loop and continue using it here.

For posterity: this will be handled separately, as a fix for #39507.

asheshvidyut · 2025-08-11T12:43:11Z

The current implementation doesn't return a running loop and creates a new loop every time we call the function _get_or_create_default_loop. Do you think it makes sense to return a running loop so that the loop created once could be reused again? cc @sergiitk @sreenithi

import asyncio
import threading

# Global variable to track the created loop
_global_loop = None
_loop_lock = threading.Lock()

def get_running_event_loop():
    global _global_loop
    
    try:
        return asyncio.get_running_loop()
    except RuntimeError:
        with _loop_lock:
            if _global_loop is not None and _global_loop.is_running():
                print("reusing existing loop")
                return _global_loop
            
            # No loop is running, so we need to create and start one
            print("creating a loop")
            loop = asyncio.new_event_loop()
            asyncio.set_event_loop(loop)
            
            # Start the loop in a daemon thread so it doesn't block the main thread
            def run_forever():
                loop.run_forever()
            
            thread = threading.Thread(target=run_forever, daemon=True)
            thread.start()
                    
            _global_loop = loop
            return loop

This reverts commit 5269922.

This reverts commit 4bbff28.

MODULE.bazel

bazel/grpc_python_deps.bzl

templates/tools/dockerfile/compile_python_314.include

…ts (#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403 Prerequisites merged before this PR: - #40317 - #40354 - #40383 Closes #40289 COPYBARA_INTEGRATE_REVIEW=#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999

sergiitk

Approved as is. There's a few comments I left, but nothing that can't be done in a follow-up PR.

sergiitk · 2025-09-08T06:10:21Z

src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py

+    # The default start method of multiprocessing in linux has changed to
+    # `forkserver` instead of `fork` from Python 3.14. This causes problems
+    # with pickling target methods that use decorators.
+    # Hence manually set to use the `fork` start method.
+    if sys.version_info >= (3, 14) and ("linux" in sys.platform):
+        multiprocessing.set_start_method("fork", force=True)


A few notes regarding this change. First, we can do this once in DynamicStubTest.setUpClass(), and force=True won't be needed. Second, we should add the same logic to tools/distrib/python/grpcio_tools/grpc_tools/test/protoc_test.py.
But this doesn't block this PR, we can fix it in a follow-up.

sergiitk · 2025-09-08T06:14:28Z

src/python/grpcio_tests/tests_aio/unit/init_test.py


+@unittest.skipIf(
+    sys.version_info >= (3, 14),
+    "Skip for Python 3.14+ until https://github.com/grpc/grpc/pull/40293 is merged",


That PR was pretty much a demo I wasn't intending to merge this specific branch. I'd rather change this to

Suggested change

"Skip for Python 3.14+ until https://github.com/grpc/grpc/pull/40293 is merged",

"Skip for Python 3.14+ until https://github.com/grpc/grpc/issues/39507 is fixed",

sergiitk · 2025-09-08T06:15:54Z

src/python/grpcio_tests/tests_aio/unit/outside_init_test.py


+@unittest.skipIf(
+    sys.version_info >= (3, 14),
+    "Skip for Python 3.14+ until https://github.com/grpc/grpc/pull/40293 is merged",


Same

Suggested change

"Skip for Python 3.14+ until https://github.com/grpc/grpc/pull/40293 is merged",

"Skip for Python 3.14+ until https://github.com/grpc/grpc/issues/39507 is fixed",

MODULE.bazel

…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999

…ters for distribtests (#40637) Backport of #40289 to v1.75.x. --- This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR #40403 Prerequisites merged before this PR: - #40317 - #40354 - #40383

…ts (grpc#40289) This PR adds Python 3.14 support for Distribtests by updating relevant docker images and distribtest build files for Linux, MacOS and Windows environments. However, artifact generation in distribtests will be enabled in the next PR grpc#40403 Prerequisites merged before this PR: - grpc#40317 - grpc#40354 - grpc#40383 Closes grpc#40289 COPYBARA_INTEGRATE_REVIEW=grpc#40289 from sreenithi:python_314_test_support 3b8bf96 PiperOrigin-RevId: 802967999

@sergiitk

… tests (grpc#40403) This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. ## Distribtests Required pre-requisite changes to enable 3.14 artifacts are covered in grpc#40289 . ## Bazel tests Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in grpc#40602 ## Basic tests The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ### 1) No running event loop for AsyncIO when run outside an async function ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in grpc#39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR grpc#40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ### 2) Pickling error from the `multiprocessing` library ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method. Closes grpc#40403 PiperOrigin-RevId: 804290760

@sergiitk

… tests (grpc#40403) This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. Required pre-requisite changes to enable 3.14 artifacts are covered in grpc#40289 . Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in grpc#40602 The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in grpc#39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR grpc#40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method. Closes grpc#40403 PiperOrigin-RevId: 804290760

@sergiitk

#40726) Backport of #40403 to v1.75.x. --- This PR enables Python 3.14 in all the different tests - Basic tests (Native Python tests), Bazel tests and Distrib tests to build Python 3.14 artifacts. In addition, it also updates all the public facing METADATA versions. ## Distribtests Required pre-requisite changes to enable 3.14 artifacts are covered in #40289 . ## Bazel tests Enabling Python 3.14 required updating the rules_python version to a more recent version that supports 3.14. This was done in #40602 ## Basic tests The following errors were caught by the Basic tests when running via Python 3.14 and resolved in this PR: ### 1) No running event loop for AsyncIO when run outside an async function ``` Traceback (most recent call last): File "src/python/grpcio/grpc/_cython/_cygrpc/aio/common.pyx.pxi", line 184, in grpc._cython.cygrpc.get_working_loop RuntimeError: no running event loop ``` This was caught by the `tests_aio.unit.outside_init_test.TestOutsideInit` and `tests_aio.unit.init_test.TestInit` tests, and was also previously reported in #39507 with the root cause. Following some investigation, the fix is being worked on by @sergiitk in PR #40293. In order to parallelize the fix and this PR, these 2 tests are currently being skipped for Python 3.14 and above. ### 2) Pickling error from the `multiprocessing` library ``` _pickle.PicklingError: Can't pickle <function _test_well_known_types at 0x7f3937eee610>: it's not the same object as tests.unit._dynamic_stubs_test._test_well_known_types when serializing dict item '_target' when serializing multiprocessing.context.Process state when serializing multiprocessing.context.Process object ``` This was caught by the `tests.unit._dynamic_stubs_test.DynamicStubTest` which runs test cases in a subprocess using the `multiprocessing` library. Error root cause: - The default start method of multiprocessing in linux has changed to `forkserver` instead of `fork` from Python 3.14. - `forkserver` has a few extra restrictions for picklability as compared to `fork` (Ref: [Python Docs](https://docs.python.org/3.14/library/multiprocessing.html#the-spawn-and-forkserver-start-methods)) - All the [test case functions](https://github.com/grpc/grpc/blob/0243842d5d10f624bf8f09f45026dd300805502f/src/python/grpcio_tests/tests/unit/_dynamic_stubs_test.py#L115) in the DynamicStubTest that are provided as `target` to the `multiprocessing.Process` use decorators. This causes problems when pickling them. Hence to resolve this, we manually set the 'start method' of `multiprocessing` to use the `fork` start method.

sreenithi added 3 commits August 6, 2025 08:47

update basictests docker image templates with Python 3.14rc1

8bda354

generate and update dockerfile hashes

1ea5c0b

add py314 configs for basic tests

653468d

sreenithi added lang/Python release notes: no Indicates if PR should not be in release notes labels Aug 6, 2025

grpc-checks bot added bloat/none per-call-memory/neutral per-channel-memory/neutral labels Aug 6, 2025

sreenithi and others added 8 commits August 7, 2025 06:46

add my fix for no running loop issue

4bbff28

temporary add python312 to presubmit tests

c3a33e2

merge sergii's fix for no aio loop issue

5269922

possible fix for multiprocess pickling issue

35ac3be

fix sanity

93002ae

add comment and fix sanity

6e3c1d7

update fix

62212e2

remove 3.12 from presubmit run

c4674ce

sreenithi marked this pull request as ready for review August 8, 2025 10:26

sreenithi requested a review from sergiitk as a code owner August 8, 2025 10:26

sreenithi added this to the Python 3.14 support milestone Aug 8, 2025

sreenithi requested a review from asheshvidyut August 8, 2025 10:28

sreenithi mentioned this pull request Aug 11, 2025

[Python][Support 3.14] Add Python 3.14 Support #40418

Closed

Update Module.bazel file

9290151

asheshvidyut reviewed Aug 11, 2025

View reviewed changes

sreenithi added 4 commits August 22, 2025 15:06

Merge branch 'master' into python_314_basictests_update

36ee365

skip tests causing aio event loop failure

e139ef4

Revert "merge sergii's fix for no aio loop issue"

2b64013

This reverts commit 5269922.

Revert "add my fix for no running loop issue"

ca64fd7

This reverts commit 4bbff28.

sreenithi self-assigned this Aug 22, 2025

Merge branch 'master' into python_314_basictests_update

4ab8fa3

try updating rules_python to use 3.14.0b2

f447b1a

sreenithi requested a review from eugeneo as a code owner September 1, 2025 12:11

sreenithi commented Sep 2, 2025

View reviewed changes

MODULE.bazel Show resolved Hide resolved

bazel/grpc_python_deps.bzl Show resolved Hide resolved

templates/tools/dockerfile/compile_python_314.include Show resolved Hide resolved

sreenithi added 6 commits September 3, 2025 12:25

Merge branch 'master' into python_314_basictests_update

ab92e54

update bazelify BUILD rules

b083cbb

enable 3.14 artifacts in distribtests

37071d2

update 3.14 in yaml file for supported python versions

70c9000

generate and update python_version.py with 3.14

a45f913

update alpine_aarch64 dockerfile with python 3.14rc1 base image

24fc559

sreenithi mentioned this pull request Sep 3, 2025

[Python][Support 3.14] Enable Python 3.14 interpreters for distribtests #40289

Closed

sreenithi changed the title ~~[Python][Support 3.14] Enable 3.14 in Python Basic tests~~ [Python][Support 3.14] Enable 3.14 in Python Basic, Bazel and Distrib tests Sep 3, 2025

Merge branch 'master' into python_314_basictests_update

383c74e

sergiitk approved these changes Sep 8, 2025

View reviewed changes

copybara-service bot closed this in ee5325b Sep 8, 2025

sreenithi mentioned this pull request Sep 8, 2025

[Backport][v1.75.x][Python][Support 3.14] Enable Python 3.14 interpreters for distribtests #40637

Merged

sreenithi mentioned this pull request Sep 17, 2025

[Backport][v1.75.x][Python][Support 3.14] Enable 3.14 supported wheels #40726

Merged

anniefrchz mentioned this pull request Oct 3, 2025

rekey anniefrchz/grpc#46

Closed

springmeyer mentioned this pull request Nov 11, 2025

Allow grpcio >=1.63.0 triton-inference-server/client#862

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Python][Support 3.14] Enable 3.14 in Python Basic, Bazel and Distrib tests#40403

[Python][Support 3.14] Enable 3.14 in Python Basic, Bazel and Distrib tests#40403
sreenithi wants to merge 25 commits intogrpc:masterfrom
sreenithi:python_314_basictests_update

sreenithi commented Aug 6, 2025 •

edited

Loading

Uh oh!

asheshvidyut Aug 11, 2025

Uh oh!

sreenithi Aug 11, 2025

Uh oh!

sergiitk Sep 8, 2025

Uh oh!

asheshvidyut commented Aug 11, 2025 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergiitk left a comment

Uh oh!

sergiitk Sep 8, 2025

Uh oh!

sergiitk Sep 8, 2025

Uh oh!

sergiitk Sep 8, 2025

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	"Skip for Python 3.14+ until https://github.com/grpc/grpc/pull/40293 is merged",
	"Skip for Python 3.14+ until https://github.com/grpc/grpc/issues/39507 is fixed",

Conversation

sreenithi commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Distribtests

Bazel tests

Basic tests

1) No running event loop for AsyncIO when run outside an async function

2) Pickling error from the multiprocessing library

Uh oh!

asheshvidyut Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

sreenithi Aug 11, 2025

Choose a reason for hiding this comment

Uh oh!

sergiitk Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

asheshvidyut commented Aug 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sergiitk left a comment

Choose a reason for hiding this comment

Uh oh!

sergiitk Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

sergiitk Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

sergiitk Sep 8, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

sreenithi commented Aug 6, 2025 •

edited

Loading

2) Pickling error from the `multiprocessing` library

asheshvidyut commented Aug 11, 2025 •

edited

Loading