Skip to content

Fix Build issue from python_include#67

Merged
angerson merged 4 commits intomasterfrom
fix-py310-build-issue
Jan 20, 2022
Merged

Fix Build issue from python_include#67
angerson merged 4 commits intomasterfrom
fix-py310-build-issue

Conversation

@sampathweb
Copy link
Contributor

Issue: Python3.10 builds fail with error -
ERROR: Analysis of target '//tensorflow/tools/pip_package:build_pip_package' failed; build aborted: Analysis of target '@local_config_python//:python_include' failed

Fixes:

  1. With the new python3.10.2, the need for special handling of pip install is not required.
  2. Map /usr/include/python to /usr/local/include/python. This is required for python 3.10 to link correctly. Its a no-op for other python version.
    when you run command -
python -c "from __future__ import print_function;import sysconfig;print(sysconfig.get_path('include'))"

it points to /usr/local/include/python3.10 whereas in other python versions it points correctly to /usr/include/python3.9. The files are in /usr/local/python3.10, so its required to map /usr/include/python3.10 to /usr/local/include/python3.10`

Testing:
Verified that with these changes the python 3.10 builds correctly by running the build locally using the staging docker container.

@sampathweb sampathweb changed the title Fix py310 build issue Fix Build issue from python_include Jan 20, 2022
@sampathweb sampathweb added the build and push to gcr.io for staging Create a staging release on gcr.io label Jan 20, 2022
@github-actions
Copy link

I pushed these containers:

  • gcr.io/tensorflow-sigs/build:67-python3.10
  • gcr.io/tensorflow-sigs/build:67-python3.9
  • gcr.io/tensorflow-sigs/build:67-python3.8
  • gcr.io/tensorflow-sigs/build:67-python3.7

Re-apply the build and push to gcr.io for staging label to rebuild and push again. This comment will only be posted once.

@angerson
Copy link
Contributor

Thanks!

@sampathweb sampathweb deleted the fix-py310-build-issue branch January 20, 2022 23:57
nitins17 added a commit that referenced this pull request Feb 16, 2022
Set up symlink for devtoolset-8

Combine Docker GCR presubmits and also push main to gcr

Commit missed files

Log in to GCR

Fix conditional, hopefully

Clarify

Add Python 3.10 support (#58)

Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work.

Upgrade gcrpio for fast build and cleanup setup

Add utilities for running release tests (#56)

This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here.

add gdb to the system packages

change to gcc 8.3.1 from centos7 for devtoolset8

fix libstdc++ symlink in devtoolset-8 environment

Undo ignoring other xml files

Update README

Deduplicate repeated messages

Squash long runfiles paths

Lock nvidia driver to 460

libtensorflow work

Fix libtensorflow script and start prelim check

Update Test Requirements to have same versions as  tf_sig_build_dockerfiles/devel.requirements.txt (#65)

* Add additional gitignore files

* Update requirements with same versions

Keep versions consistent with  tf_sig_build_dockerfiles/devel.requirements.txt

Cleanup

Fix Build issue from `python_include` (#67)

* Remove Python 3.10 pip special handling

* Link usr/include to usr/local/include

* Update location of python include

* Update setup.python.sh

Assorted changes -- see details

- Remove installation of nvidia-profiler, which depends on libcuda1,
which ultimately installs an nvidia driver package, which we don't want
because we're running in docker, in which the drivers are mounted. I
hope nvidia-profiler isn't necessary for anything important; otherwise
we'll need to synchronize driver versions between the containers and VM
images.
- Add less, colordiff and a newer version of clang-format
- Add code_check_changed_files, which is intended to replace the
"incremental" parts of ci_sanity. Still a work in progress because we
need to decide on valuable configurations (clang-format and pylint
cannot be run the same way as we have them configured internally and
currently have a lot of findings)
- Add code_check_full, which is intended to replace the "across entire
code base" parts of ci_sanity. I rewrote many of the clunkier tests.
Still a work in progress because we must verify that the changed tests
will still fail.
- Fix bad "bazel test " expansion for libtensorflow
- Fix bad chmod for libtensoflow repacker

Change libtensorflow config values to fix target selection

Fix a typo in venv installation

(Thanks to reedwm)

Remove extra lines

(Thanks again to reedwm)

Clarify ctrl-s warning

Correctly remove extra test filters

Make it possible to run isolated pip tests

More work on code checks

Fix a typo

Clean up code check full

Remove clang-format

Cleanup changed_files and move one to full

Add a missing test

Clean up and fix code_check_full

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

change to devtoolset-9 and gcc 9.3.1 for manylinux2014
nitins17 added a commit that referenced this pull request Mar 2, 2022
Set up symlink for devtoolset-8

Combine Docker GCR presubmits and also push main to gcr

Commit missed files

Log in to GCR

Fix conditional, hopefully

Clarify

Add Python 3.10 support (#58)

Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work.

Upgrade gcrpio for fast build and cleanup setup

Add utilities for running release tests (#56)

This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here.

add gdb to the system packages

change to gcc 8.3.1 from centos7 for devtoolset8

fix libstdc++ symlink in devtoolset-8 environment

Undo ignoring other xml files

Update README

Deduplicate repeated messages

Squash long runfiles paths

Lock nvidia driver to 460

libtensorflow work

Fix libtensorflow script and start prelim check

Update Test Requirements to have same versions as  tf_sig_build_dockerfiles/devel.requirements.txt (#65)

* Add additional gitignore files

* Update requirements with same versions

Keep versions consistent with  tf_sig_build_dockerfiles/devel.requirements.txt

Cleanup

Fix Build issue from `python_include` (#67)

* Remove Python 3.10 pip special handling

* Link usr/include to usr/local/include

* Update location of python include

* Update setup.python.sh

Assorted changes -- see details

- Remove installation of nvidia-profiler, which depends on libcuda1,
which ultimately installs an nvidia driver package, which we don't want
because we're running in docker, in which the drivers are mounted. I
hope nvidia-profiler isn't necessary for anything important; otherwise
we'll need to synchronize driver versions between the containers and VM
images.
- Add less, colordiff and a newer version of clang-format
- Add code_check_changed_files, which is intended to replace the
"incremental" parts of ci_sanity. Still a work in progress because we
need to decide on valuable configurations (clang-format and pylint
cannot be run the same way as we have them configured internally and
currently have a lot of findings)
- Add code_check_full, which is intended to replace the "across entire
code base" parts of ci_sanity. I rewrote many of the clunkier tests.
Still a work in progress because we must verify that the changed tests
will still fail.
- Fix bad "bazel test " expansion for libtensorflow
- Fix bad chmod for libtensoflow repacker

Change libtensorflow config values to fix target selection

Fix a typo in venv installation

(Thanks to reedwm)

Remove extra lines

(Thanks again to reedwm)

Clarify ctrl-s warning

Correctly remove extra test filters

Make it possible to run isolated pip tests

More work on code checks

Fix a typo

Clean up code check full

Remove clang-format

Cleanup changed_files and move one to full

Add a missing test

Clean up and fix code_check_full

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

change to devtoolset-9 and gcc 9.3.1 for manylinux2014
nitins17 added a commit that referenced this pull request Mar 15, 2022
Set up symlink for devtoolset-8

Combine Docker GCR presubmits and also push main to gcr

Commit missed files

Log in to GCR

Fix conditional, hopefully

Clarify

Add Python 3.10 support (#58)

Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work.

Upgrade gcrpio for fast build and cleanup setup

Add utilities for running release tests (#56)

This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here.

add gdb to the system packages

change to gcc 8.3.1 from centos7 for devtoolset8

fix libstdc++ symlink in devtoolset-8 environment

Undo ignoring other xml files

Update README

Deduplicate repeated messages

Squash long runfiles paths

Lock nvidia driver to 460

libtensorflow work

Fix libtensorflow script and start prelim check

Update Test Requirements to have same versions as  tf_sig_build_dockerfiles/devel.requirements.txt (#65)

* Add additional gitignore files

* Update requirements with same versions

Keep versions consistent with  tf_sig_build_dockerfiles/devel.requirements.txt

Cleanup

Fix Build issue from `python_include` (#67)

* Remove Python 3.10 pip special handling

* Link usr/include to usr/local/include

* Update location of python include

* Update setup.python.sh

Assorted changes -- see details

- Remove installation of nvidia-profiler, which depends on libcuda1,
which ultimately installs an nvidia driver package, which we don't want
because we're running in docker, in which the drivers are mounted. I
hope nvidia-profiler isn't necessary for anything important; otherwise
we'll need to synchronize driver versions between the containers and VM
images.
- Add less, colordiff and a newer version of clang-format
- Add code_check_changed_files, which is intended to replace the
"incremental" parts of ci_sanity. Still a work in progress because we
need to decide on valuable configurations (clang-format and pylint
cannot be run the same way as we have them configured internally and
currently have a lot of findings)
- Add code_check_full, which is intended to replace the "across entire
code base" parts of ci_sanity. I rewrote many of the clunkier tests.
Still a work in progress because we must verify that the changed tests
will still fail.
- Fix bad "bazel test " expansion for libtensorflow
- Fix bad chmod for libtensoflow repacker

Change libtensorflow config values to fix target selection

Fix a typo in venv installation

(Thanks to reedwm)

Remove extra lines

(Thanks again to reedwm)

Clarify ctrl-s warning

Correctly remove extra test filters

Make it possible to run isolated pip tests

More work on code checks

Fix a typo

Clean up code check full

Remove clang-format

Cleanup changed_files and move one to full

Add a missing test

Clean up and fix code_check_full

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

change to devtoolset-9 and gcc 9.3.1 for manylinux2014

change cachebuster value for ml2014 remote cache

change to new libstdcxx abi for devtoolset-9

change cachebuster value to use the new libstdcxx abi

link against nonshared44 in devtoolset-9

update the cachebuster value

change CACHEBUSTER value for gpu builds

remove redudant commands during build environment setup

change cachbuster variable name for gpu builds

store manylinux2014 cache in a different location
nitins17 added a commit that referenced this pull request Mar 15, 2022
Set up symlink for devtoolset-8

Combine Docker GCR presubmits and also push main to gcr

Commit missed files

Log in to GCR

Fix conditional, hopefully

Clarify

Add Python 3.10 support (#58)

Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work.

Upgrade gcrpio for fast build and cleanup setup

Add utilities for running release tests (#56)

This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here.

add gdb to the system packages

change to gcc 8.3.1 from centos7 for devtoolset8

fix libstdc++ symlink in devtoolset-8 environment

Undo ignoring other xml files

Update README

Deduplicate repeated messages

Squash long runfiles paths

Lock nvidia driver to 460

libtensorflow work

Fix libtensorflow script and start prelim check

Update Test Requirements to have same versions as  tf_sig_build_dockerfiles/devel.requirements.txt (#65)

* Add additional gitignore files

* Update requirements with same versions

Keep versions consistent with  tf_sig_build_dockerfiles/devel.requirements.txt

Cleanup

Fix Build issue from `python_include` (#67)

* Remove Python 3.10 pip special handling

* Link usr/include to usr/local/include

* Update location of python include

* Update setup.python.sh

Assorted changes -- see details

- Remove installation of nvidia-profiler, which depends on libcuda1,
which ultimately installs an nvidia driver package, which we don't want
because we're running in docker, in which the drivers are mounted. I
hope nvidia-profiler isn't necessary for anything important; otherwise
we'll need to synchronize driver versions between the containers and VM
images.
- Add less, colordiff and a newer version of clang-format
- Add code_check_changed_files, which is intended to replace the
"incremental" parts of ci_sanity. Still a work in progress because we
need to decide on valuable configurations (clang-format and pylint
cannot be run the same way as we have them configured internally and
currently have a lot of findings)
- Add code_check_full, which is intended to replace the "across entire
code base" parts of ci_sanity. I rewrote many of the clunkier tests.
Still a work in progress because we must verify that the changed tests
will still fail.
- Fix bad "bazel test " expansion for libtensorflow
- Fix bad chmod for libtensoflow repacker

Change libtensorflow config values to fix target selection

Fix a typo in venv installation

(Thanks to reedwm)

Remove extra lines

(Thanks again to reedwm)

Clarify ctrl-s warning

Correctly remove extra test filters

Make it possible to run isolated pip tests

More work on code checks

Fix a typo

Clean up code check full

Remove clang-format

Cleanup changed_files and move one to full

Add a missing test

Clean up and fix code_check_full

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

change to devtoolset-9 and gcc 9.3.1 for manylinux2014

change cachebuster value for ml2014 remote cache

change to new libstdcxx abi for devtoolset-9

change cachebuster value to use the new libstdcxx abi

link against nonshared44 in devtoolset-9

update the cachebuster value

change CACHEBUSTER value for gpu builds

remove redudant commands during build environment setup

change cachbuster variable name for gpu builds

store manylinux2014 cache in a different location

amend comment for accuracy
angerson pushed a commit that referenced this pull request Mar 21, 2022
Set up symlink for devtoolset-8

Combine Docker GCR presubmits and also push main to gcr

Commit missed files

Log in to GCR

Fix conditional, hopefully

Clarify

Add Python 3.10 support (#58)

Adds Python 3.10 support to the containers. Python 3.10 changes some library behavior and, for now, needs an alternative installation method to work.

Upgrade gcrpio for fast build and cleanup setup

Add utilities for running release tests (#56)

This adds the dependencies and notably bazelrc config options to run TensorFlow's Nightly and Release tests, which I've been working on replicating on internal CI. I still have documentation and migration work to do, but the major portion of the support work is here.

add gdb to the system packages

change to gcc 8.3.1 from centos7 for devtoolset8

fix libstdc++ symlink in devtoolset-8 environment

Undo ignoring other xml files

Update README

Deduplicate repeated messages

Squash long runfiles paths

Lock nvidia driver to 460

libtensorflow work

Fix libtensorflow script and start prelim check

Update Test Requirements to have same versions as  tf_sig_build_dockerfiles/devel.requirements.txt (#65)

* Add additional gitignore files

* Update requirements with same versions

Keep versions consistent with  tf_sig_build_dockerfiles/devel.requirements.txt

Cleanup

Fix Build issue from `python_include` (#67)

* Remove Python 3.10 pip special handling

* Link usr/include to usr/local/include

* Update location of python include

* Update setup.python.sh

Assorted changes -- see details

- Remove installation of nvidia-profiler, which depends on libcuda1,
which ultimately installs an nvidia driver package, which we don't want
because we're running in docker, in which the drivers are mounted. I
hope nvidia-profiler isn't necessary for anything important; otherwise
we'll need to synchronize driver versions between the containers and VM
images.
- Add less, colordiff and a newer version of clang-format
- Add code_check_changed_files, which is intended to replace the
"incremental" parts of ci_sanity. Still a work in progress because we
need to decide on valuable configurations (clang-format and pylint
cannot be run the same way as we have them configured internally and
currently have a lot of findings)
- Add code_check_full, which is intended to replace the "across entire
code base" parts of ci_sanity. I rewrote many of the clunkier tests.
Still a work in progress because we must verify that the changed tests
will still fail.
- Fix bad "bazel test " expansion for libtensorflow
- Fix bad chmod for libtensoflow repacker

Change libtensorflow config values to fix target selection

Fix a typo in venv installation

(Thanks to reedwm)

Remove extra lines

(Thanks again to reedwm)

Clarify ctrl-s warning

Correctly remove extra test filters

Make it possible to run isolated pip tests

More work on code checks

Fix a typo

Clean up code check full

Remove clang-format

Cleanup changed_files and move one to full

Add a missing test

Clean up and fix code_check_full

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

Update docs and create experimental RBE configs

Update docs and create experimental RBE configs

Update dependencies to 2.9.0.dev

Update Go API installation guide for TensorFlow 2.8.0 (#74)

Clarify usage of nightly commit

Fix mistaken 'test' command

change to devtoolset-9 and gcc 9.3.1 for manylinux2014

change cachebuster value for ml2014 remote cache

change to new libstdcxx abi for devtoolset-9

change cachebuster value to use the new libstdcxx abi

link against nonshared44 in devtoolset-9

update the cachebuster value

change CACHEBUSTER value for gpu builds

remove redudant commands during build environment setup

change cachbuster variable name for gpu builds

store manylinux2014 cache in a different location

amend comment for accuracy
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build and push to gcr.io for staging Create a staging release on gcr.io

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants