@NancyLi1013 Where do you see these status?

The ParquetSharp above shows that the arrow port does build from scratch for x64-windows-static, x64-linux and x64-osx on GitHub Actions / Azure.

I'm not sure if Apache Arrow supports x86 (e.g. PyArrow has no wheels for x86, https://pypi.org/project/pyarrow/#files). And I've never tried Arm64.

The real question is why does the vcpkg CI display a green build when it contains the following?
Error: Building package arrow:x64-linux failed with: BUILD_FAILED

Hi @GPSnoopy

I got these info from ci.baseline.txt
https://github.com/microsoft/vcpkg/blob/master/scripts/ci.baseline.txt#L86-L88

If arrow doesn't support arm64, linux and osx, x86, we need to add vcpkg_fail_port_install(ON_ARCH "x86" "arm64" ON_TARGET "linux" "osx") to the top of portfile.cmake.

If not, we need to fix the problems on these triplets.

As for the CI question you mentioned above, since we have set arrow:x64-linux=fail in ci.baseline.txt.

The CI check results depend on the status of triplets for the ports in ci.baseline.txt and build results. If the two statuses are consistent, the final check will show pass, namely display a green.

@NancyLi1013 I've removed the fail status for both linux and osx. I've also replaced the adhoc check on arch with vcpkg_fail_port_install, such that the build will fail with x86, arm and arm64.

The CI is properly showing the failures, as you explained. Now, we know that these targets all pass when run via ParquetSharp CI. So what's different with vcpkg CI?

x64-linux

This one really baffles me. It seems to go wrong in Arrow's FindBrotli.

-- Checking for modules 'libbrotlicommon;libbrotlienc;libbrotlidec'
--   Found libbrotlicommon, version 1.0.7
--   Found libbrotlienc, version 1.0.7
--   Found libbrotlidec, version 1.0.7
-- Configuring incomplete, errors occurred!

CMake Error at /mnt/_work/1/s/downloads/tools/cmake-3.17.2-linux/cmake-3.17.2-Linux-x86_64/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find Brotli (missing: BROTLI_COMMON_LIBRARY BROTLI_ENC_LIBRARY
  BROTLI_DEC_LIBRARY)
Call Stack (most recent call first):
  /mnt/_work/1/s/downloads/tools/cmake-3.17.2-linux/cmake-3.17.2-Linux-x86_64/share/cmake-3.17/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  cmake_modules/FindBrotli.cmake:107 (find_package_handle_standard_args)
  /mnt/_work/1/s/scripts/buildsystems/vcpkg.cmake:329 (_find_package)
  cmake_modules/ThirdpartyToolchain.cmake:156 (find_package)
  cmake_modules/ThirdpartyToolchain.cmake:874 (resolve_dependency)
  CMakeLists.txt:458 (include)

However I cannot reproduce this error. It works absolutely fine when run on my machine and via GitHub Actions (see PR link in the description); note that both are running Ubuntu 18.04. This is what I get in my logs instead:

-- Checking for modules 'libbrotlicommon;libbrotlienc;libbrotlidec'
--   Found libbrotlicommon, version 1.0.7
--   Found libbrotlienc, version 1.0.7
--   Found libbrotlidec, version 1.0.7
-- Found Brotli: /home/gpsnoopy/Development/vcpkg.tanguyf/packages/brotli_x64-linux/debug/lib/libbrotlicommon-static.a

Not being able to reproduce the issue locally is making this incredibly difficult to figure out. Any ideas?

x64-osx

[1/95] cd /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp && /usr/bin/bison -d -o /Volumes/data/buildtrees/thrift/x64-osx-dbg/compiler/cpp/thrift/thrifty.cc /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy
FAILED: compiler/cpp/thrift/thrifty.cc compiler/cpp/thrift/thrifty.hh 
cd /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp && /usr/bin/bison -d -o /Volumes/data/buildtrees/thrift/x64-osx-dbg/compiler/cpp/thrift/thrifty.cc /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy
/Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy:1.1-5: invalid directive: `%code'
/Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy:1.7-14: syntax error, unexpected identifier

I believe the issue here is the Thrift dependency on Bison. The default version included in macOS is super old (2007 IIRC). This is why ParquetSharp CI does brew install bison on macOS before running vcpkg.

What's the best way to solve this? Sort of feels like this should be in the CI rather than in Thrift port.

x64-osx

[1/95] cd /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp && /usr/bin/bison -d -o /Volumes/data/buildtrees/thrift/x64-osx-dbg/compiler/cpp/thrift/thrifty.cc /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy
FAILED: compiler/cpp/thrift/thrifty.cc compiler/cpp/thrift/thrifty.hh 
cd /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp && /usr/bin/bison -d -o /Volumes/data/buildtrees/thrift/x64-osx-dbg/compiler/cpp/thrift/thrifty.cc /Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy
/Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy:1.1-5: invalid directive: `%code'
/Volumes/data/work/1/s/buildtrees/thrift/src/401ebbec17-573d9ca574/compiler/cpp/src/thrift/thrifty.yy:1.7-14: syntax error, unexpected identifier

I believe the issue here is the Thrift dependency on Bison. The default version included in macOS is super old (2007 IIRC). This is why ParquetSharp CI does brew install bison on macOS before running vcpkg.

I can confirm that this issue is because the default Bison is too old on macOS. This is why a newer version of Bison is downloaded by vcpkg on Windows:

See also my comment here: #7533 (comment)

/azp run

I can reproduce the CI failure locally by using the vcpkg ci command: ./vcpkg ci x64-linux --x-xunit="$PWD/xml-results/x64-linux.xml" --exclude=${SKIPLIST} --binarycaching --purge-tombstones (after setting up SKIPLIST to skip most things except arrow and its dependencies). However, running ./vcpkg install arrow:x64-linux on the same machine from a separate clean checkout does run successfully.

It looks like the problem is that when the ci build runs, the packages directory gets cleared after each package is installed, whereas running the install command doesn’t do that, and the logs show that arrow is using Brotli from in the packages directory rather than the installed directory:

Found Brotli: /mnt/c/Users/adree/gr/vcpkg/packages/brotli_x64-linux/debug/lib/libbrotlicommon-static.a

I’m not very familiar with CMake, but FindBrotli.cmake in arrow appears to use pkg-config to locate Brotli with the pkg_check_modules function. However, the pkg-config file for Brotli at vcpkg/installed/x64-linux/lib/pkgconfig/libbrotlicommon.pc for example references the Brotli directories within the vcpkg/packages directory that no longer exist when using the ci command:

prefix=/mnt/c/Users/adree/gr/vcpkg/packages/brotli_x64-linux
exec_prefix=/mnt/c/Users/adree/gr/vcpkg/packages/brotli_x64-linux
libdir=${prefix}/lib
includedir=${prefix}/include

Name: libbrotlicommon
URL: https://github.com/google/brotli
Description: Brotli common dictionary library
Version: 1.0.7
Libs: -L${libdir} -lbrotlicommon
Cflags: -I${includedir}

So it seems that because vcpkg moves files from packages into installed, the pkg-config files are no longer correct and arrow then looks in the wrong place for Brotli.

I guess the fix here might be to patch FindBrotli.cmake in the arrow port so it uses a different method to locate the Brotli libraries, but it seems a bit sad to have to do that to work around vcpkg behaviour. Would it make sense for vcpkg to understand pkg-config files and fix paths in them when it moves files from packages into installed?

@NancyLi1013 Thanks to @adamreeve changes, the x64-linux CI build is now working.

I've marked back x64-osx as still failing until a proper solution for Bison tooling on macOS can be devised. Open to ideas.

In the meantime, I'm happy for the PR to be merged if you are.

Hi @NancyLi1013, thanks for the review and the patch.

The osx failure seems to be unrelated to this PR, I think it's safe to merge.

Thanks for your kindly reminder.
The failure on x64-osx is not related with this PR. We will try to fix this.
Once fixed, I will rerun this PR.

All features have passed with the following triplets:

• x64-windows
• x64-windows-static

Note: This port only supports x64.

x64-linux has passed too. Or did I miss something?

Let us all pray to the merger gods now :-)

LGTM now, @GPSnoopy thanks for this PR.