tensorflow-osx is failing in CI #21758
Description
This issue is tracking investigation as to why tensorflow and tensorflow-cc are broken on our MacOS CI machines. Consider a recent CI run: https://dev.azure.com/vcpkg/public/_build/results?buildId=63779
I was able to reproduce this locally, tensorflow fails with:
bion@Billys-MBP vcpkg % ./vcpkg install tensorflow tensorflow-cc
Computing installation plan...
The following packages will be built and installed:
tensorflow[core]:x64-osx -> 2.6.0
tensorflow-cc[core]:x64-osx -> 2.6.0
* tensorflow-common[core]:x64-osx -> 2.6.0
Additional packages (*) will be modified to complete this operation.
Detecting compiler hash for triplet x64-osx...
Restored 0 packages from /Users/bion/.cache/vcpkg/archives in 1.243 ms. Use --debug to see more details.
Starting package 1/3: tensorflow-common:x64-osx
Building package tensorflow-common[core]:x64-osx...
-- Installing: /Users/bion/vcpkg/packages/tensorflow-common_x64-osx/share/tensorflow-common/copyright
-- Performing post-build validation
-- Performing post-build validation done
Stored binary cache: /Users/bion/.cache/vcpkg/archives/bf/bff52bffca58f369d3f2db4f408db568032660656907661c9b668adf94fc4109.zip
Installing package tensorflow-common[core]:x64-osx...
Elapsed time for package tensorflow-common:x64-osx: 306.5 ms
Starting package 2/3: tensorflow:x64-osx
Building package tensorflow[core]:x64-osx...
-- Downloading https://github.com/bazelbuild/bazel/releases/download/4.1.0/bazel-4.1.0-darwin-x86_64 -> bazel-4.1.0-darwin-x86_64...
-- Installing: /Users/bion/vcpkg/downloads/tools/bazel/4.1.0-darwin/bazel
CMake Error at scripts/cmake/vcpkg_execute_required_process.cmake:127 (message):
Command failed: /usr/bin/python3 -m pip install --user -U --force-reinstall numpy
Working Directory: /Users/bion/vcpkg/buildtrees/tensorflow
Error code: 1
See logs for more information:
/Users/bion/vcpkg/buildtrees/tensorflow/prerequesits-pip-x64-osx-out.log
/Users/bion/vcpkg/buildtrees/tensorflow/prerequesits-pip-x64-osx-err.log
Call Stack (most recent call first):
installed/x64-osx/share/tensorflow-common/tensorflow-common.cmake:54 (vcpkg_execute_required_process)
ports/tensorflow/portfile.cmake:7 (include)
scripts/ports.cmake:142 (include)
Error: Building package tensorflow:x64-osx failed with: BUILD_FAILED
When I run
/usr/bin/python3 -m pip install --user -U --force-reinstall numpy
directly outside of vcpkg I indeed get build failures attempting to compile numpy. However, I got a warning message:
ERROR: Could not build wheels for numpy which use PEP 517 and cannot be installed directly
WARNING: You are using pip version 20.2.3; however, version 21.3.1 is available.
You should consider upgrading via the '/Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip' command.
After /Library/Developer/CommandLineTools/usr/bin/python3 -m pip install --upgrade pip, tensorflow appears to install OK.
I'm not sure how to fix this, since upgrading pip seems to be a "system modifying" action we don't want vcpkg to do. I could just manually upgrade pip on all the osx test hardware, but that seems to be just masking the problem.
We could add an extra help message, but ideally there would be some way to update pip in a private "user" way like we try to install numpy in the first place...
/cc @jgehw