Skip to content

[Backport][v1.75.x][Python] aio: skip grpc/aio shutdown if py interpreter is finalizing#40649

Merged
sergiitk merged 1 commit intogrpc:v1.75.xfrom
sergiitk:backport-1.75-fix/aio/shutdown
Sep 17, 2025
Merged

[Backport][v1.75.x][Python] aio: skip grpc/aio shutdown if py interpreter is finalizing#40649
sergiitk merged 1 commit intogrpc:v1.75.xfrom
sergiitk:backport-1.75-fix/aio/shutdown

Conversation

@sergiitk
Copy link
Member

@sergiitk sergiitk commented Sep 9, 2025

Backport of #40447 to v1.75.x.

This PR changes the logic of shutdown_grpc_aio to skip _actual_aio_shutdown python interpreter is already being finalized (cleaning up resources, destroying objects, preparing for program exit, etc). _actual_aio_shutdown involves PollerCompletionQueue shutdown, followed by core grpc_shutdown API call.

Reasoning:

  1. During finalizations, in come cases resources we're accessing may already be freed, and the order is not deterministic. Some of the resources being unloaded prior the _actual_aio_shutdown call: _global_aio_state, AsyncIOEngine enum, or even python libraries like sys. This leads to errors like AttributeError: 'NoneType' object has no attribute 'POLLER'.
  2. PollerCompletionQueue.shutdown() will try to wait on its poller thread to finish gracefully. In py3.14, PythonFinalizationError is raised when Thread.join() is called during finalization. I think the logic here is similar to (1): these threads may have already been deallocated.

Note that in some cases users were able to prevent _actual_aio_shutdown from being called by manually calling init_grpc_aio prior to initializing any grpc objects. This resulted in an incorrect positive refcount, which prevents _actual_aio_shutdown from being run. Before the above finalization check was added this side-effect was sometimes misused to avoid deadlock on finialization (#22365).

This PR:

@sergiitk sergiitk self-assigned this Sep 9, 2025
@sergiitk sergiitk added lang/Python release notes: yes Indicates if PR needs to be in release notes labels Sep 9, 2025
@sergiitk sergiitk requested a review from sreenithi September 9, 2025 21:56
@sergiitk
Copy link
Member Author

sergiitk commented Sep 9, 2025

Note: do not merge until 1.75.0 is out. This is intended for 1.75.1.

…rpc#40447)

This PR changes the logic of `shutdown_grpc_aio` to skip `_actual_aio_shutdown` python interpreter is already [being finalized](https://docs.python.org/3.14/glossary.html#term-interpreter-shutdown) (cleaning up resources, destroying objects, preparing for program exit, etc). `_actual_aio_shutdown` involves `PollerCompletionQueue` shutdown, followed by core [`grpc_shutdown`](https://grpc.github.io/grpc/core/grpc_8h.html#a35f55253e80714c17f4f3a0657e06f1b) API call.

Reasoning:
1. During finalizations, in come cases resources we're accessing may already be freed, and the order is not deterministic. Some of the resources being unloaded prior the  `_actual_aio_shutdown` call: `_global_aio_state`, `AsyncIOEngine` enum, or even python libraries like `sys`. This leads to errors like `AttributeError: 'NoneType' object has no attribute 'POLLER'`.
2. `PollerCompletionQueue.shutdown()` will try to wait on its poller thread to finish gracefully. In py3.14, `PythonFinalizationError` is raised when `Thread.join()` is called during finalization. I think the logic here is similar to (1): these threads may have already been deallocated.

Note that in some cases users were able to prevent `_actual_aio_shutdown` from being called by manually calling `init_grpc_aio` prior to initializing any grpc objects.  This resulted in an incorrect positive refcount, which prevents `_actual_aio_shutdown` from being run. Before the above finalization check was added this side-effect was sometimes misused to avoid deadlock on finialization (grpc#22365).

This PR:
- Fixes grpc#39520
- Fixes grpc#22365
- Fixes grpc#38679
- Fixes grpc#33342
- Fixes grpc#36655

Closes grpc#40447

COPYBARA_INTEGRATE_REVIEW=grpc#40447 from sergiitk:fix/aio/shutdown 11114f6
PiperOrigin-RevId: 804971756
@sergiitk sergiitk force-pushed the backport-1.75-fix/aio/shutdown branch from eaf018a to 5326c8c Compare September 16, 2025 22:58
@sergiitk sergiitk marked this pull request as ready for review September 16, 2025 22:59
@sergiitk sergiitk merged commit ff24d38 into grpc:v1.75.x Sep 17, 2025
57 of 58 checks passed
@sergiitk sergiitk deleted the backport-1.75-fix/aio/shutdown branch September 17, 2025 17:35
@sergiitk sergiitk added this to the Python 3.14 support milestone Sep 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

lang/Python release notes: yes Indicates if PR needs to be in release notes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants