Replace distutils.sysconfig with sysconfig by jjyao · Pull Request #28392 · grpc/grpc

jjyao · 2021-12-20T21:58:58Z

This is essentially tensorflow/tensorflow#51776 since python_configure.bzl is modified from tensorflow.

I basically copied two PRs from there: tensorflow/tensorflow#51782 and tensorflow/tensorflow#51865.

@donnadionne

linux-foundation-easycla · 2021-12-20T21:59:01Z

The committers are authorized under a signed CLA.

✅ Jiajun Yao (a275032)

jjyao · 2021-12-20T22:04:48Z

Tag @lidizheng @gnossen as well.

lidizheng · 2021-12-20T23:16:38Z

Docker Hub is down now, I will retrigger the tests when it's back-up.

jjyao · 2021-12-21T15:54:21Z

@lidizheng I think docker hub is back-up now :)

lidizheng · 2021-12-21T17:02:24Z

INFO: Repository local_config_python instantiated at:
  /var/local/git/grpc/WORKSPACE:5:10: in <toplevel>
  /var/local/git/grpc/bazel/grpc_deps.bzl:448:21: in grpc_deps
  /var/local/git/grpc/bazel/grpc_python_deps.bzl:61:21: in grpc_python_deps
Repository rule python_configure defined at:
  /var/local/git/grpc/third_party/py/python_configure.bzl:365:35: in <toplevel>
ERROR: An error occurred during the fetch of repository 'local_config_python':
   Traceback (most recent call last):
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 347, column 35, in _python_autoconf_impl
		_create_single_version_package(
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 304, column 45, in _create_single_version_package
		python_include = _get_python_include(repository_ctx, python_bin)
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 253, column 13, in _get_python_include
		_execute(
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 62, column 14, in _execute
		_fail("\n".join([
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 35, column 9, in _fail
		fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Unable to find Python headers for /usr/bin/python2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: /usr/local/include/python2.7/Python.h does not exist.
Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
ERROR: Error fetching repository: Traceback (most recent call last):
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 347, column 35, in _python_autoconf_impl
		_create_single_version_package(
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 304, column 45, in _create_single_version_package
		python_include = _get_python_include(repository_ctx, python_bin)
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 253, column 13, in _get_python_include
		_execute(
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 62, column 14, in _execute
		_fail("\n".join([
	File "/var/local/git/grpc/third_party/py/python_configure.bzl", line 35, column 9, in _fail
		fail("%sPython Configuration Error:%s %s\n" % (red, no_color, msg))
Error in fail: Python Configuration Error: Unable to find Python headers for /usr/bin/python2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: /usr/local/include/python2.7/Python.h does not exist.
Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
INFO: Repository com_google_libprotobuf_mutator instantiated at:
  /var/local/git/grpc/WORKSPACE:7:20: in <toplevel>
  /var/local/git/grpc/bazel/grpc_deps.bzl:528:21: in grpc_test_only_deps
Repository rule http_archive defined at:
  /root/.cache/bazel/_bazel_root/d2dc70c3d9da3fab488ba0dcbbd35051/external/bazel_tools/tools/build_defs/repo/http.bzl:336:31: in <toplevel>
ERROR: /var/local/git/grpc/src/python/grpcio/grpc/_cython/BUILD.bazel:25:12: //src/python/grpcio/grpc/_cython:cygrpc.so depends on @local_config_python//:python_headers in repository @local_config_python which failed to fetch. no such package '@local_config_python//': Python Configuration Error: Unable to find Python headers for /usr/bin/python2
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AssertionError: /usr/local/include/python2.7/Python.h does not exist.
Are Python headers installed? Try installing python-dev or python3-dev on Debian-based systems. Try python-devel or python3-devel on Redhat-based systems.
ERROR: Analysis of target '//examples/python/multiprocessing:prime_proto_pb2_grpc' failed; build aborted: Analysis failed
INFO: Elapsed time: 28.827s

There might be some Python 2 residual causing tests to fail (there is also an ongoing setuptools issue). If you are comfortable with Bazel, do you think you can take a peak at it?

acxz · 2021-12-21T17:12:30Z

@jjyao can you fix the corresponding error msg?

            python_bin + ".) " + "Is distutils installed? " +
            _HEADERS_HELP

to

            python_bin + ".) " + _HEADERS_HELP

as seen in my PR.

This is just for consistency since distutils does not need to be installed as sysconfig is part of the default python installation now.

However it is not for python2 I believe. Thus for the tests that are failing regarding python2 we may need to have a conditional that runs the old code with distutils.

jjyao · 2021-12-21T18:01:12Z

Oh, does grpc support python2? I thought it's >= 3.6 based on https://github.com/grpc/grpc/tree/master/src/python/grpcio.

lidizheng · 2021-12-21T18:06:09Z

Nah, this looks like a script used by C/C++ build. I can take a look later.

jjyao · 2021-12-21T18:13:10Z

Nah, this looks like a script used by C/C++ build. I can take a look later.

Thanks. I'm not familiar with the grpc build system so it would be great if you can provide some help here.

lidizheng · 2021-12-21T18:18:39Z

Link to an existing issue: #28026

acxz · 2021-12-21T22:51:51Z

@lidizheng can you clarify on what is needed for this PR to be merged?

Is getting the CI to pass a hard requirement? Just want to know since quite a few downstream projects are stuck at this issue and would like to know when this can be rolled out.

lidizheng · 2021-12-21T23:03:51Z

During last weekend, upstream setuptools broke the Python build, I'm working on two fixes #28393 and #28397.

Yes, getting the CI pass is a hard requirement.

@acxz I understand this is a headache for Bazel + 3.10. For code rollout, do you need a patch release or fix at head is fine?

acxz · 2021-12-21T23:21:50Z

Yes, getting the CI pass is a hard requirement.

Got it. If I get some bandwidth I'll see if I can help with #28026 and eliminating all python 2 code.

The way we use grpc in our Bazel, a fix at head is just what we need.

lidizheng · 2021-12-28T18:18:19Z

@jjyao Build related issues have been fixed, the Python 2 issue in Bazel is also fixed. Can you rebase this PR?

jjyao · 2021-12-28T18:25:25Z

@lidizheng Rebased.

lidizheng

Thanks for making this update!

lidizheng · 2021-12-28T21:31:15Z

@gnossen PTAL.

gnossen

Thanks for the contribution!

add patch for newer setuptools, can be removed once grpc 1.44 is release Why are these changes needed? With grpc updated to 1.43, one of the patches is not needed. Patch needed when building locally for newer setuptools version. See grpc/grpc#28392 for more details. Also needed as a prereq to #21221

add patch for newer setuptools, can be removed once grpc 1.44 is release Why are these changes needed? With grpc updated to 1.43, one of the patches is not needed. Patch needed when building locally for newer setuptools version. See grpc/grpc#28392 for more details. Also needed as a prereq to ray-project#21221

Replace distutils.sysconfig with sysconfig

a275032

lidizheng added kokoro:run lang/Python release notes: no Indicates if PR should not be in release notes labels Dec 20, 2021

lidizheng requested review from gnossen and lidizheng December 20, 2021 22:52

lidizheng assigned gnossen and lidizheng Dec 20, 2021

grpc-kokoro removed the kokoro:run label Dec 20, 2021

grpc-checks bot added the bloat/none label Dec 20, 2021

grpc-checks bot added the perf-change/none label Dec 20, 2021

lidizheng added the kokoro:force-run label Dec 21, 2021

grpc-kokoro removed the kokoro:force-run label Dec 21, 2021

acxz mentioned this pull request Dec 21, 2021

remove deprecated distutils calls #28395

Closed

Replace distutils.sysconfig with sysconfig

d00e7da

lidizheng added the kokoro:run label Dec 21, 2021

grpc-kokoro removed the kokoro:run label Dec 21, 2021

lidizheng mentioned this pull request Dec 22, 2021

Remove Python2 from python_configure.bzl #28406

Merged

Merge branch 'master' of github.com:grpc/grpc into jjyao/distutils

957a9d4

lidizheng added the kokoro:run label Dec 28, 2021

grpc-kokoro removed the kokoro:run label Dec 28, 2021

lidizheng approved these changes Dec 28, 2021

View reviewed changes

gnossen approved these changes Dec 29, 2021

View reviewed changes

lidizheng merged commit dbe73c9 into grpc:master Dec 29, 2021

jjyao deleted the jjyao/distutils branch December 29, 2021 22:09

jjyao mentioned this pull request Jan 3, 2022

Add support for Python 3.10 ray-project/ray#21221

Merged

6 tasks

acxz mentioned this pull request Jan 25, 2022

update grpc to 1.43 ray-project/ray#21866

Merged

6 tasks

Conversation

jjyao commented Dec 20, 2021

Uh oh!

linux-foundation-easycla bot commented Dec 20, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjyao commented Dec 20, 2021

Uh oh!

lidizheng commented Dec 20, 2021

Uh oh!

jjyao commented Dec 21, 2021

Uh oh!

lidizheng commented Dec 21, 2021

Uh oh!

acxz commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jjyao commented Dec 21, 2021

Uh oh!

lidizheng commented Dec 21, 2021

Uh oh!

jjyao commented Dec 21, 2021

Uh oh!

lidizheng commented Dec 21, 2021

Uh oh!

acxz commented Dec 21, 2021

Uh oh!

lidizheng commented Dec 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

acxz commented Dec 21, 2021

Uh oh!

lidizheng commented Dec 28, 2021

Uh oh!

jjyao commented Dec 28, 2021

Uh oh!

lidizheng left a comment

Choose a reason for hiding this comment

Uh oh!

lidizheng commented Dec 28, 2021

Uh oh!

gnossen left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

linux-foundation-easycla bot commented Dec 20, 2021 •

edited

Loading

acxz commented Dec 21, 2021 •

edited

Loading

lidizheng commented Dec 21, 2021 •

edited

Loading