Add Python 3.10 support by sampathweb · Pull Request #58 · tensorflow/build

sampathweb · 2021-12-03T01:09:49Z

No description provided.

github-actions · 2021-12-03T20:43:43Z

I pushed these containers:

gcr.io/tensorflow-sigs/build:58-python3.10
gcr.io/tensorflow-sigs/build:58-python3.9
gcr.io/tensorflow-sigs/build:58-python3.8
gcr.io/tensorflow-sigs/build:58-python3.7

Re-apply the build and push to gcr.io for staging label to rebuild and push again. This comment will only be posted once.

sampathweb · 2021-12-08T01:49:41Z

Python 3.10 build succeeded and uploaded to PyPi.
https://pypi.org/project/tf-nightly-cpu/2.8.0.dev20211208/#files
https://pypi.org/project/tf-nightly/2.8.0.dev20211208/

angerson · 2021-12-08T20:38:11Z

tf_sig_build_dockerfiles/setup.python.sh

-curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
-python3 get-pip.py
+if [[ "$VERSION" == "python3.10" ]]; then
+  # Python 3.10 pip reference is broken for pip 21.3 (pypa/pip#10647)


Change this to a regular link, please

angerson · 2021-12-08T20:42:40Z

tf_sig_build_dockerfiles/setup.python.sh

 source ~/.bashrc
 VERSION=$1
 REQUIREMENTS=$2
+echo $PATH


Remove this

angerson · 2021-12-08T20:42:50Z

tf_sig_build_dockerfiles/setup.python.sh

 # Disable the cache dir to save image space, and install packages
-python3 -m pip install --no-cache-dir --upgrade pip
 python3 -m pip install --no-cache-dir -r $REQUIREMENTS -U
+python3 -m pip list


Remove this

angerson · 2021-12-08T21:24:29Z

tf_sig_build_dockerfiles/setup.python.sh

+if [[ "$VERSION" == "python3.10" ]]; then
+  # Python 3.10 pip reference is broken for pip 21.3 (pypa/pip#10647)
+  # TODO(rameshsampath): Remove once Python 3.10 works with latest pip
+  python3 -m ensurepip


What happens if you replace this entire install-pip section with just this?

python3 -m ensurepip python3 -m pip install --no-cache-dir --upgrade pip

I ran a quick test and didn't see any obvious problems. Can you give more details about the issue that's being avoided?

In one of the earlier builds, where I had the kept the setup that was there in the docker I ran into one of two problems:

No module named pip (https://github.com/tensorflow/build/runs/4420931846?check_suite_focus=true#step:5:1893)

It installs the packages, but not in the right place and hence not able to find it when trying to use it.

Also, Updating Ensurepip doesn't work for Python 3.8 since we are updating system python and gives an error message: https://github.com/tensorflow/build/runs/4463376066?check_suite_focus=true#step:5:1204

angerson · 2021-12-08T22:04:38Z

tf_sig_build_dockerfiles/setup.python.sh

+  # Python 3.10 pip reference is broken for pip 21.3 (https://github.com/pypa/pip/issues/10647)
+  # TODO(rameshsampath): Remove once Python 3.10 works with latest pip
+  python3 -m ensurepip
+  python3 -m pip install pip~=21.2.0


Why does this need to be pinned to 21.2.x? Can you go into more detail?

Current release is 21.3 which doesn't work with Python 3.10, so just going one minor version back. I can add that to the text.

I mean, what specifically doesn't work in these containers when the pip version is 21.3?

When I install pip version 21.3, it doesn't install the packages. Here's the problem I run into -

(py310) rameshsampath@rsamdev:~/temp$ docker run -it --rm gcr.io/tensorflow-sigs/build:58-python3.10 tf-docker / > pip -V pip 21.3.1 from /usr/lib/python3.10/dist-packages/pip (python 3.10) tf-docker / > pip install numpy Collecting numpy Downloading numpy-1.21.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (15.9 MB) |████████████████████████████████| 15.9 MB 8.4 MB/s Installing collected packages: numpy Successfully installed numpy-1.21.4 WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv tf-docker / > python Python 3.10.0 (default, Oct 4 2021, 22:09:55) [GCC 9.3.0] on linux Type "help", "copyright", "credits" or "license" for more information. >>> import numpy Traceback (most recent call last): File "<stdin>", line 1, in <module> ModuleNotFoundError: No module named 'numpy' tf-docker / > pip list Package Version ----------------------- -------------------- certifi 2019.11.28 chardet 3.0.4 dbus-python 1.2.16 grpcio 1.38.1 idna 2.8 pip 21.3.1 PyGObject 3.36.0 python-apt 2.0.0+ubuntu0.20.4.6 requests 2.22.0 requests-unixsocket 0.2.0 screen-resolution-extra 0.0.0 setuptools 57.4.0 termcolor 1.1.0 urllib3 1.25.8 wrapt 1.12.1 xkit 0.0.0

Here it uses the latest pip 21.3, doesn't fail on requirements install, but doesn't show up in the pip list. This is why I had to downgrade to pip 21.2.x

Thanks! That's very helpful.

I am hoping to come back to this in Q1 and remove this workaround.

angerson · 2021-12-10T20:54:03Z

I'm going to clean up the scripts and comments after merging this.

angerson · 2021-12-10T21:55:05Z

Done, see b0f7b46.

Set up symlink for devtoolset-8 Combine Docker GCR presubmits and also push main to gcr Commit missed files Log in to GCR Fix conditional, hopefully Clarify Add Python 3.10 support (#58) Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work. Upgrade gcrpio for fast build and cleanup setup Add utilities for running release tests (#56) This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here. add gdb to the system packages change to gcc 8.3.1 from centos7 for devtoolset8 fix libstdc++ symlink in devtoolset-8 environment Undo ignoring other xml files Update README Deduplicate repeated messages Squash long runfiles paths Lock nvidia driver to 460 libtensorflow work Fix libtensorflow script and start prelim check Update Test Requirements to have same versions as tf_sig_build_dockerfiles/devel.requirements.txt (#65) * Add additional gitignore files * Update requirements with same versions Keep versions consistent with tf_sig_build_dockerfiles/devel.requirements.txt Cleanup Fix Build issue from `python_include` (#67) * Remove Python 3.10 pip special handling * Link usr/include to usr/local/include * Update location of python include * Update setup.python.sh Assorted changes -- see details - Remove installation of nvidia-profiler, which depends on libcuda1, which ultimately installs an nvidia driver package, which we don't want because we're running in docker, in which the drivers are mounted. I hope nvidia-profiler isn't necessary for anything important; otherwise we'll need to synchronize driver versions between the containers and VM images. - Add less, colordiff and a newer version of clang-format - Add code_check_changed_files, which is intended to replace the "incremental" parts of ci_sanity. Still a work in progress because we need to decide on valuable configurations (clang-format and pylint cannot be run the same way as we have them configured internally and currently have a lot of findings) - Add code_check_full, which is intended to replace the "across entire code base" parts of ci_sanity. I rewrote many of the clunkier tests. Still a work in progress because we must verify that the changed tests will still fail. - Fix bad "bazel test " expansion for libtensorflow - Fix bad chmod for libtensoflow repacker Change libtensorflow config values to fix target selection Fix a typo in venv installation (Thanks to reedwm) Remove extra lines (Thanks again to reedwm) Clarify ctrl-s warning Correctly remove extra test filters Make it possible to run isolated pip tests More work on code checks Fix a typo Clean up code check full Remove clang-format Cleanup changed_files and move one to full Add a missing test Clean up and fix code_check_full Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command change to devtoolset-9 and gcc 9.3.1 for manylinux2014

Set up symlink for devtoolset-8 Combine Docker GCR presubmits and also push main to gcr Commit missed files Log in to GCR Fix conditional, hopefully Clarify Add Python 3.10 support (#58) Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work. Upgrade gcrpio for fast build and cleanup setup Add utilities for running release tests (#56) This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here. add gdb to the system packages change to gcc 8.3.1 from centos7 for devtoolset8 fix libstdc++ symlink in devtoolset-8 environment Undo ignoring other xml files Update README Deduplicate repeated messages Squash long runfiles paths Lock nvidia driver to 460 libtensorflow work Fix libtensorflow script and start prelim check Update Test Requirements to have same versions as tf_sig_build_dockerfiles/devel.requirements.txt (#65) * Add additional gitignore files * Update requirements with same versions Keep versions consistent with tf_sig_build_dockerfiles/devel.requirements.txt Cleanup Fix Build issue from `python_include` (#67) * Remove Python 3.10 pip special handling * Link usr/include to usr/local/include * Update location of python include * Update setup.python.sh Assorted changes -- see details - Remove installation of nvidia-profiler, which depends on libcuda1, which ultimately installs an nvidia driver package, which we don't want because we're running in docker, in which the drivers are mounted. I hope nvidia-profiler isn't necessary for anything important; otherwise we'll need to synchronize driver versions between the containers and VM images. - Add less, colordiff and a newer version of clang-format - Add code_check_changed_files, which is intended to replace the "incremental" parts of ci_sanity. Still a work in progress because we need to decide on valuable configurations (clang-format and pylint cannot be run the same way as we have them configured internally and currently have a lot of findings) - Add code_check_full, which is intended to replace the "across entire code base" parts of ci_sanity. I rewrote many of the clunkier tests. Still a work in progress because we must verify that the changed tests will still fail. - Fix bad "bazel test " expansion for libtensorflow - Fix bad chmod for libtensoflow repacker Change libtensorflow config values to fix target selection Fix a typo in venv installation (Thanks to reedwm) Remove extra lines (Thanks again to reedwm) Clarify ctrl-s warning Correctly remove extra test filters Make it possible to run isolated pip tests More work on code checks Fix a typo Clean up code check full Remove clang-format Cleanup changed_files and move one to full Add a missing test Clean up and fix code_check_full Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command change to devtoolset-9 and gcc 9.3.1 for manylinux2014 change cachebuster value for ml2014 remote cache change to new libstdcxx abi for devtoolset-9 change cachebuster value to use the new libstdcxx abi link against nonshared44 in devtoolset-9 update the cachebuster value change CACHEBUSTER value for gpu builds remove redudant commands during build environment setup change cachbuster variable name for gpu builds store manylinux2014 cache in a different location

Set up symlink for devtoolset-8 Combine Docker GCR presubmits and also push main to gcr Commit missed files Log in to GCR Fix conditional, hopefully Clarify Add Python 3.10 support (#58) Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work. Upgrade gcrpio for fast build and cleanup setup Add utilities for running release tests (#56) This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here. add gdb to the system packages change to gcc 8.3.1 from centos7 for devtoolset8 fix libstdc++ symlink in devtoolset-8 environment Undo ignoring other xml files Update README Deduplicate repeated messages Squash long runfiles paths Lock nvidia driver to 460 libtensorflow work Fix libtensorflow script and start prelim check Update Test Requirements to have same versions as tf_sig_build_dockerfiles/devel.requirements.txt (#65) * Add additional gitignore files * Update requirements with same versions Keep versions consistent with tf_sig_build_dockerfiles/devel.requirements.txt Cleanup Fix Build issue from `python_include` (#67) * Remove Python 3.10 pip special handling * Link usr/include to usr/local/include * Update location of python include * Update setup.python.sh Assorted changes -- see details - Remove installation of nvidia-profiler, which depends on libcuda1, which ultimately installs an nvidia driver package, which we don't want because we're running in docker, in which the drivers are mounted. I hope nvidia-profiler isn't necessary for anything important; otherwise we'll need to synchronize driver versions between the containers and VM images. - Add less, colordiff and a newer version of clang-format - Add code_check_changed_files, which is intended to replace the "incremental" parts of ci_sanity. Still a work in progress because we need to decide on valuable configurations (clang-format and pylint cannot be run the same way as we have them configured internally and currently have a lot of findings) - Add code_check_full, which is intended to replace the "across entire code base" parts of ci_sanity. I rewrote many of the clunkier tests. Still a work in progress because we must verify that the changed tests will still fail. - Fix bad "bazel test " expansion for libtensorflow - Fix bad chmod for libtensoflow repacker Change libtensorflow config values to fix target selection Fix a typo in venv installation (Thanks to reedwm) Remove extra lines (Thanks again to reedwm) Clarify ctrl-s warning Correctly remove extra test filters Make it possible to run isolated pip tests More work on code checks Fix a typo Clean up code check full Remove clang-format Cleanup changed_files and move one to full Add a missing test Clean up and fix code_check_full Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command Update docs and create experimental RBE configs Update docs and create experimental RBE configs Update dependencies to 2.9.0.dev Update Go API installation guide for TensorFlow 2.8.0 (#74) Clarify usage of nightly commit Fix mistaken 'test' command change to devtoolset-9 and gcc 9.3.1 for manylinux2014 change cachebuster value for ml2014 remote cache change to new libstdcxx abi for devtoolset-9 change cachebuster value to use the new libstdcxx abi link against nonshared44 in devtoolset-9 update the cachebuster value change CACHEBUSTER value for gpu builds remove redudant commands during build environment setup change cachbuster variable name for gpu builds store manylinux2014 cache in a different location amend comment for accuracy

Add Python 3.10 support

87bac2b

angerson mentioned this pull request Dec 3, 2021

Add Python 3.10 Docker Build #51

Closed

sampathweb added the build and push to gcr.io for staging Create a staging release on gcr.io label Dec 3, 2021

sampathweb added 8 commits December 4, 2021 22:14

Use venv

3d0bec7

Use Virtual Env

c175c3b

Use Virtual Env

2c8416e

Use Virtual Env

af24ca6

Use Virtual Env

5983bb4

Use Virtual Env

88436d2

Use Virtual Env

70acc6c

Add comments for Python 3.10

0080eb1

sampathweb requested a review from angerson December 7, 2021 23:32

sampathweb marked this pull request as ready for review December 7, 2021 23:32

sampathweb requested a review from perfinion as a code owner December 7, 2021 23:32

Update JAX Lib for Python 3.10

1f72cdf

angerson reviewed Dec 8, 2021

View reviewed changes

sampathweb added 2 commits December 8, 2021 13:48

Update Pip

89dec32

Update Pip

d6e3728

angerson suggested changes Dec 8, 2021

View reviewed changes

sampathweb added 2 commits December 9, 2021 14:03

update pip

0d166d7

update pip

961d0c6

angerson merged commit 85bc99a into master Dec 10, 2021

angerson deleted the python3.10 branch December 10, 2021 20:55

Conversation

sampathweb commented Dec 3, 2021

Uh oh!

github-actions bot commented Dec 3, 2021

Uh oh!

sampathweb commented Dec 8, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

angerson commented Dec 10, 2021

Uh oh!

angerson commented Dec 10, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sampathweb commented Dec 8, 2021 •

edited

Loading