[Python] Upgrade Pytype (Part - 1) by asheshvidyut · Pull Request #39816 · grpc/grpc

asheshvidyut · 2025-06-10T06:29:14Z

Description

Add a new script to run an upgraded version of PyType
Keeping the old script / version running as is.

This is Part 1 PR of upgrading PyType in which only one folder is added as input. Final inputs in the second PR will be similar to -

grpc/setup.cfg

Lines 21 to 28 in 8342a10

    
           [pytype] 
        
           inputs = 
        
               src/python/grpcio/grpc/experimental 
        
               src/python/grpcio/grpc 
        
               src/python/grpcio_tests/tests_aio 
        
               src/python/grpcio_observability/grpc_observability 
        
               examples/python/auth 
        
               examples/python/helloworld

Follow up PR will have other folders added.

Testing

Script runs in CI. Hence testing included in CI.

setup_updated.cfg

asheshvidyut · 2025-06-11T07:22:41Z

setup_updated.cfg

+    src/python/grpcio/grpc/beta
+    src/python/grpcio/grpc/__init__.py
+    src/python/grpcio/grpc/_simple_stubs.py
+output = ~/.cache/pytype


Sanity tests run in parallel. Hence this is required.

grpc/tools/run_tests/sanity/sanity_tests.yaml

Lines 30 to 31 in 62facbe

- script: tools/distrib/check_pytype.sh

- script: tools/distrib/check_pytype_updated.sh

tools/distrib/check_pytype_updated.sh

sergiitk · 2025-06-18T02:25:26Z

tools/distrib/check_pytype_updated.sh

+source $VIRTUALENV/bin/activate
+
+python3 -m pip install pytype==2024.10.11
+python3 -m pytype --keep-going -j "$JOBS" --strict-import --config "setup_updated.cfg"


same here

Suggested change

python3 -m pytype --keep-going -j "$JOBS" --strict-import --config "setup_updated.cfg"

pytype --keep-going -j auto --strict-import --config "setup_updated.cfg"

upd: -j auto eliminates the need for nproc

also - we should just set it in the config

I think strict-import is strict_import in the config too, and keep-going is keep_going

sergiitk · 2025-06-18T02:27:24Z

tools/distrib/check_pytype_updated.sh

+
+JOBS=$(nproc) || JOBS=4
+
+mkdir -p ~/.cache/pytype


it seems to create it on its own, do we need this?

$ python3 -m pytype --keep-going -j "$JOBS" --strict-import --config "setup_updated.cfg" Computing dependencies Analyzing 18 sources with 13 local dependencies ninja: Entering directory `../../.cache/pytype' ... $ realpath ../../.cache/pytype /Users/sergiitk/.cache/pytype

setup_updated.cfg

sergiitk · 2025-06-18T02:57:03Z

setup_updated.cfg

+# NOTE(lidiz) Adding examples one by one due to pytype aggressive error:
+# ninja: error: build.ninja:178: multiple rules generate helloworld_pb2.pyi [-w dupbuild=err]
+# TODO(xuanwn): include all files in src/python/grpcio/grpc
+[pytype]


looks like pytype supports toml! https://github.com/google/pytype/tree/main?tab=readme-ov-file#config-file

I've converted this with (minor changes aside)

$ uvx 'ini2toml[lite]' setup_updated.cfg -o setup_updated.toml Profile not explicitly set, using 'best_effort'.

Result:

[tool.pytype] inputs = [ "src/python/grpcio/grpc", ] exclude = [ "**/*_pb2.py", "src/python/grpcio/grpc/framework", "src/python/grpcio/grpc/aio", "src/python/grpcio/grpc/beta", "src/python/grpcio/grpc/__init__.py", "src/python/grpcio/grpc/_simple_stubs.py", ] output = "~/.cache/pytype" disable = [ "import-error", "module-attr", "attribute-error", ]

We can place it to black.toml (which we should probably rename at some point). Works just fine!

$ pytype --config=black.toml --keep-going -j "$JOBS" --strict-import Computing dependencies Analyzing 18 sources with 13 local dependencies ninja: Entering directory `../../.cache/pytype' [48/48] check grpc.experimental.aio.__init__ Leaving directory '../../.cache/pytype' Success: no errors found

I have converted setup_updated.cfg to pytype.toml. I would prefer not merging it with black.toml as both a different tools and have seperate configuration, might give better readability and maintainability

A reason not to use pytype.toml is because it's what picked automatically - I'd like to avoid that for period we're migrating to the updated tooling.

A reason to use black.toml is that it's already contains settings for other tools (like isort). It's a normal pattern for all tolling setting to be in one pyproject.toml file - but we can't do it right now without breaking setup.py.

But I see why black.toml may be very confusing. I mentioned that " (which we should probably rename at some point)."

Should we rename it now then? Like grpc-style-config.toml?

sergiitk · 2025-06-18T03:14:03Z

src/python/grpcio/grpc/_observability.py

-def create_server_call_tracer_factory_option(xds: bool) -> ChannelArgumentType:
+def create_server_call_tracer_factory_option(
+    xds: bool,
+) -> ServerCallTracerFactoryOption:


Maybe just Tuple[ChannelArgumentType, ...]?

I think the main problem was that we declared return as ChannelArgumentType (which is tuple[str, Any]), but ended up returning a tuple containing ChannelArgumentType.

And , ... covers an empty tuple, thought it may imply there may be multiple options.
Alternatively we could do Union[Tuple[ChannelArgumentType], Tuple[()]] - if we want to be VERY clear that it's either a single-element tuple with an option or an empty one.

I'm fine either way.

This (Tuple[ChannelArgumentType, ...]) is invalid for python3.7 and currently our CI has check_pytype.sh which runs on 3.7.

https://btx.cloud.google.com/invocations/5e69e776-ff9c-4e83-909a-6cba2b3caaef/targets/github%2Fgrpc%2Frun_tests%2Fsanity_linux_dbg_native%2Ftools%2Fdistrib%2Fcheck_pytype.sh;config=default/tests

src/python/grpcio/grpc/_typing.py

Co-authored-by: Sergii Tkachenko <sergiitk@google.com>

sergiitk · 2025-06-18T23:23:55Z

tools/distrib/check_pytype_updated.sh

+JOBS=$(nproc) || JOBS=4
+


JOBS=$(nproc) || JOBS=4

can be removed now

sergiitk · 2025-06-18T23:33:47Z

tools/distrib/check_pytype_updated.sh

+source $VIRTUALENV/bin/activate
+
+pip install pytype==2024.10.11
+pytype -j auto --config=pytype.toml


Let's move auto to the config, but provide --output=~/.cache/pytype as an argument.

jobs = 'auto'

Reasons:

We do want to autodetect cpu number even when run pytype locally

No reason to write grpc-only output to ~/.cache/pytype when running pytype locally

sreenithi · 2025-06-19T08:55:59Z

Just a note for reference, I was initially considering if this new script needs to get added to sanitize.sh as well. But given that this script is similar to check_pytype.sh, which doesn't exist in sanitize.sh currently, this file is probably not needed to be added there too.

sergiitk · 2025-06-23T19:17:28Z

Good catch, @sreenithi. Since we have this test passing now, I think we should include it into sanitize.sh. Let's discuss it tomorrow.

### Description * Add a new script to run an upgraded version of PyType * Keeping the old script / version running as is. * This is Part 1 PR of upgrading PyType in which only one folder is added as input. Final inputs in the second PR will be similar to - https://github.com/grpc/grpc/blob/8342a109aece77f04147b0a82a4e09ba25466e03/setup.cfg#L21-L28 * Follow up PR will have other folders added. ### Testing * Script runs in CI. Hence testing included in CI. Closes grpc#39816 COPYBARA_INTEGRATE_REVIEW=grpc#39816 from asheshvidyut:upgrade-pytype-part-01 68955ae PiperOrigin-RevId: 773260007

github-actions bot added lang/Python lang/core labels Jun 10, 2025

asheshvidyut added the release notes: yes Indicates if PR needs to be in release notes label Jun 10, 2025

grpc-checks bot added bloat/none per-call-memory/neutral per-channel-memory/neutral labels Jun 10, 2025

Upgrade Pytype Part - 1

5e28b3e

asheshvidyut force-pushed the upgrade-pytype-part-01 branch from cf70568 to 5e28b3e Compare June 11, 2025 02:24

asheshvidyut added 7 commits June 11, 2025 08:47

keep old script

8ae5882

using pyton3.11

34472ab

revert

e854f65

using a different cache dir for pytype updated

62facbe

creating the new cache folder

de1c39a

create a new type

d150544

fix black

f155ac8

asheshvidyut commented Jun 11, 2025

View reviewed changes

setup_updated.cfg Outdated Show resolved Hide resolved

asheshvidyut commented Jun 11, 2025

View reviewed changes

tools/distrib/check_pytype_updated.sh Outdated Show resolved Hide resolved

asheshvidyut added kokoro:run and removed kokoro:run labels Jun 11, 2025

asheshvidyut marked this pull request as ready for review June 11, 2025 15:55

asheshvidyut requested review from sergiitk and sreenithi June 11, 2025 15:55

sergiitk reviewed Jun 18, 2025

View reviewed changes

asheshvidyut and others added 6 commits June 18, 2025 09:45

Update tools/distrib/check_pytype_updated.sh

fb51123

Co-authored-by: Sergii Tkachenko <sergiitk@google.com>

Update tools/distrib/check_pytype_updated.sh

d7042b5

Co-authored-by: Sergii Tkachenko <sergiitk@google.com>

Update tools/distrib/check_pytype_updated.sh

2ca8173

Co-authored-by: Sergii Tkachenko <sergiitk@google.com>

Update tools/distrib/check_pytype_updated.sh

ecfbe5c

Co-authored-by: Sergii Tkachenko <sergiitk@google.com>

remove licence and todo

0ab89d0

some pr comment fixes

197c58d

asheshvidyut added 10 commits June 18, 2025 07:50

removed unsed file

50c5bfb

simplify type hint

0aaa6a3

using virutalenv

9ba1627

remove ugrade deps

5c9f4d5

add output

a51f1c5

fix toml

289cdbb

fix isort

c3f9365

fix pytype on 3.7

be4497e

fix tuple -> Tuple

5c1520c

merge master

948acb6

asheshvidyut requested a review from sergiitk June 18, 2025 15:00

sergiitk reviewed Jun 18, 2025

View reviewed changes

asheshvidyut added 3 commits June 19, 2025 05:43

address comments

5765451

remove output from grpc-stype-config.toml

e3425ca

removed unwated jobs

68955ae

asheshvidyut requested a review from sergiitk June 19, 2025 00:24

sergiitk approved these changes Jun 19, 2025

View reviewed changes

copybara-service bot closed this in dd32555 Jun 19, 2025

asheshvidyut added this to the Python Type Hints Support milestone Aug 7, 2025

	[pytype]
	inputs =
	src/python/grpcio/grpc/experimental
	src/python/grpcio/grpc
	src/python/grpcio_tests/tests_aio
	src/python/grpcio_observability/grpc_observability
	examples/python/auth
	examples/python/helloworld

	- script: tools/distrib/check_pytype.sh
	- script: tools/distrib/check_pytype_updated.sh

	python3 -m pytype --keep-going -j "$JOBS" --strict-import --config "setup_updated.cfg"
	pytype --keep-going -j auto --strict-import --config "setup_updated.cfg"

Conversation

asheshvidyut commented Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Testing

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

asheshvidyut Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sergiitk Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sergiitk Jun 18, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sreenithi commented Jun 19, 2025

Uh oh!

sergiitk commented Jun 23, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

asheshvidyut commented Jun 10, 2025 •

edited

Loading

asheshvidyut Jun 18, 2025 •

edited

Loading

sergiitk Jun 18, 2025 •

edited

Loading

sergiitk Jun 18, 2025 •

edited

Loading