Skip to content

Update vcpkg tool, add windows-arm64, and download rather than build on some POSIX#20838

Merged
BillyONeal merged 4 commits intomicrosoft:masterfrom
BillyONeal:download-script
Oct 30, 2021
Merged

Update vcpkg tool, add windows-arm64, and download rather than build on some POSIX#20838
BillyONeal merged 4 commits intomicrosoft:masterfrom
BillyONeal:download-script

Conversation

@BillyONeal
Copy link
Copy Markdown
Member

@BillyONeal BillyONeal commented Oct 19, 2021

@BillyONeal BillyONeal added info:internal category:infrastructure Pertaining to the CI/Testing infrastrucutre category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly labels Oct 19, 2021
@BillyONeal
Copy link
Copy Markdown
Member Author

Resolves #16850

I think

@autoantwort
Copy link
Copy Markdown
Contributor

autoantwort commented Oct 19, 2021

I think you should change a central file to trigger a world rebuild.

@BillyONeal
Copy link
Copy Markdown
Member Author

I think you should change a central file to trigger a world rebuild.

I'm going to trigger a CI run for that :)

@BillyONeal
Copy link
Copy Markdown
Member Author

Something is messed up because it isn't honoring --exclude, for instance libressl is being built even though it is set to skip in ci.baseline.txt. Debugging....

@BillyONeal
Copy link
Copy Markdown
Member Author

Something is messed up because it isn't honoring --exclude, for instance libressl is being built even though it is set to skip in ci.baseline.txt. Debugging....

It was microsoft/vcpkg-tool#224

@BillyONeal
Copy link
Copy Markdown
Member Author

@BillyONeal
Copy link
Copy Markdown
Member Author

/azp run

@azure-pipelines
Copy link
Copy Markdown

Azure Pipelines successfully started running 1 pipeline(s).

@BillyONeal
Copy link
Copy Markdown
Member Author

@BillyONeal
Copy link
Copy Markdown
Member Author

@BillyONeal
Copy link
Copy Markdown
Member Author

@autoantwort Most of the failures now are failing the new pkgconfig check. :/

@autoantwort
Copy link
Copy Markdown
Contributor

@autoantwort Most of the failures now are failing the new pkgconfig check. :/

Yeah that was to be expected. I can create a PR for that.

@BillyONeal
Copy link
Copy Markdown
Member Author

@autoantwort Most of the failures now are failing the new pkgconfig check. :/

Yeah that was to be expected. I can create a PR for that.

OK can you see the failures there? (I never know how auth behaves for Pipelines)

I'll take a look when I'm back at my desk with the big machine on Sunday.

@autoantwort
Copy link
Copy Markdown
Contributor

OK can you see the failures there? (I never know how auth behaves for Pipelines)

Yeah I can see the logs.

I'll take a look when I'm back at my desk with the big machine on Sunday.

I plan to do it tomorrow 👍

@Neumann-A
Copy link
Copy Markdown
Contributor

Hmm needs further refinement.
print2(Color::warning, "There should be no pkgconfig directories outside of lib and debug/lib.\n");
is wrong. share is also allowed.

            return pkgconfig_parent_name == "share" &&
                   !Util::any_of(fs.read_lines(path, VCPKG_LINE_INFO),
                                 [](const auto& line) { return Strings::starts_with(line, "Libs"); });

is too restrictive. The default template contains in general empty Libs: and/or Libs.private:

@autoantwort
Copy link
Copy Markdown
Contributor

I currently getting:

parallels@debian-gnu-linux-10:~/git_projects/vcpkg$ ./bootstrap-vcpkg.sh 
Downloading vcpkg-glibc...
Telemetry
---------
vcpkg collects usage data in order to help us improve your experience.
The data collected by Microsoft is anonymous.
You can opt-out of telemetry by re-running the bootstrap-vcpkg script with -disableMetrics,
passing --disable-metrics to vcpkg on the command line,
or by setting the VCPKG_DISABLE_METRICS environment variable.

Read more about vcpkg telemetry at docs/about/privacy.md
parallels@debian-gnu-linux-10:~/git_projects/vcpkg$ ./vcpkg 
bash: ./vcpkg: cannot execute binary file: Exec format error

I assume it downloads a x64 binary for arm64?

@autoantwort
Copy link
Copy Markdown
Contributor

And with the new tools the installation of libpng is broken:

Installing package libpng[core]:x64-linux...
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/lib/pkgconfig/libpng.pc: Invalid argument
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/lib/libpng.a: Invalid argument
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/tools/libpng/libpng-config: Invalid argument
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/debug/lib/pkgconfig/libpng.pc: Invalid argument
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/debug/lib/libpng.a: Invalid argument
failed: /home/leander/vcpkg/vcpkg_installed/x64-linux/debug/tools/libpng/libpng-config: Invalid argument
Elapsed time for package libpng:x64-linux: 9.179 ms

Which causes a lot of packages to fail in the linux pipeline.

@BillyONeal
Copy link
Copy Markdown
Member Author

I assume it downloads a x64 binary for arm64?

Yeah... we have a lot of x64-on-linux assumptions and it was my understanding that that wasn't a design goal, but you trying there suggests that we need to keep it working.

And with the new tools the installation of libpng is broken:

For some reason the copy_symlink equivalent isn't working, looking into it now.

@BillyONeal BillyONeal changed the title Update vcpkg tool +arm64, and download rather than building on POSIX Update vcpkg tool, add windows-arm64, and download rather than build on some POSIX Oct 27, 2021
@BillyONeal
Copy link
Copy Markdown
Member Author

No, I need to figure out how to get arm binaries.

After some research the only folks I can see practically doing this use physical arm64 hardware and I don't expect to get that soon, so I've restored building from source unless we can prove that the system is amd64.

@BillyONeal
Copy link
Copy Markdown
Member Author

Over Discord @autoantwort confirmed that this version works (by downloading the sources) on their Raspberry Pi.

@BillyONeal BillyONeal added the depends:different-pr This PR or Issue depends on a PR which has been filed label Oct 28, 2021
PASSING, REMOVE FROM FAIL LIST: libopenmpt:x64-linux (.\scripts\ci.baseline.txt)
PASSING, REMOVE FROM FAIL LIST: libopenmpt:x64-osx (.\scripts\ci.baseline.txt)
@BillyONeal
Copy link
Copy Markdown
Member Author

BillyONeal commented Oct 28, 2021

PASSING, REMOVE FROM FAIL LIST: libopenmpt:x64-linux (.\scripts\ci.baseline.txt)
PASSING, REMOVE FROM FAIL LIST: libopenmpt:x64-osx (.\scripts\ci.baseline.txt)

Done.

REGRESSION: avcpp:x64-linux. If expected, add avcpp:x64-linux=fail to .\scripts\ci.baseline.txt.
REGRESSION: avcpp:x64-osx. If expected, add avcpp:x64-osx=fail to .\scripts\ci.baseline.txt.

There should be no pkgconfig directories outside of lib and debug/lib.
The following misplaced pkgconfig files were found:

    /mnt/vcpkg-ci/packages/avcpp_x64-linux/share/pkgconfig/libavcpp.pc

You can move the pkgconfig files with the following commands:

    file(INSTALL "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig/name.pc" DESTINATION "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")
    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")

#21046

REGRESSION: cminpack:x64-linux. If expected, add cminpack:x64-linux=fail to .\scripts\ci.baseline.txt.

There should be no pkgconfig directories outside of lib and debug/lib.
The following misplaced pkgconfig files were found:

    /mnt/vcpkg-ci/packages/cminpack_x64-linux/lib64/pkgconfig/cminpackld.pc
    /mnt/vcpkg-ci/packages/cminpack_x64-linux/lib64/pkgconfig/cminpack.pc
    /mnt/vcpkg-ci/packages/cminpack_x64-linux/lib64/pkgconfig/cminpacks.pc
    /mnt/vcpkg-ci/packages/cminpack_x64-linux/debug/lib64/pkgconfig/cminpackld.pc
    /mnt/vcpkg-ci/packages/cminpack_x64-linux/debug/lib64/pkgconfig/cminpack.pc
    /mnt/vcpkg-ci/packages/cminpack_x64-linux/debug/lib64/pkgconfig/cminpacks.pc

You can move the pkgconfig files with the following commands:

    file(INSTALL "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig/name.pc" DESTINATION "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")
    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")

#21047

REGRESSION: libftdi:arm64-windows. If expected, add libftdi:arm64-windows=fail to .\scripts\ci.baseline.txt.
REGRESSION: libftdi:x64-windows-static-md. If expected, add libftdi:x64-windows-static-md=fail to .\scripts\ci.baseline.txt.
REGRESSION: libftdi:x64-windows-static. If expected, add libftdi:x64-windows-static=fail to .\scripts\ci.baseline.txt.
REGRESSION: libftdi:x64-windows. If expected, add libftdi:x64-windows=fail to .\scripts\ci.baseline.txt.

The following misplaced pkgconfig files were found:

    D:/packages/libftdi_x64-windows/debug/lib64/pkgconfig/libftdi.pc
    D:/packages/libftdi_x64-windows/debug/lib64/pkgconfig/libftdipp.pc
    D:/packages/libftdi_x64-windows/lib64/pkgconfig/libftdi.pc
    D:/packages/libftdi_x64-windows/lib64/pkgconfig/libftdipp.pc

You can move the pkgconfig files with the following commands:

    file(INSTALL "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig/name.pc" DESTINATION "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")
    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")

#21045

REGRESSION: miniz:arm-uwp. If expected, add miniz:arm-uwp=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:arm64-windows. If expected, add miniz:arm64-windows=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-linux. If expected, add miniz:x64-linux=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-osx. If expected, add miniz:x64-osx=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-uwp. If expected, add miniz:x64-uwp=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-windows-static-md. If expected, add miniz:x64-windows-static-md=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-windows-static. If expected, add miniz:x64-windows-static=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x64-windows. If expected, add miniz:x64-windows=fail to .\scripts\ci.baseline.txt.
REGRESSION: miniz:x86-windows. If expected, add miniz:x86-windows=fail to .\scripts\ci.baseline.txt.

This was #21025

REGRESSION: osg-qt:x64-linux. If expected, add osg-qt:x64-linux=fail to .\scripts\ci.baseline.txt.

The following misplaced pkgconfig files were found:

    /mnt/vcpkg-ci/packages/osg-qt_x64-linux/lib64/pkgconfig/openscenegraph-osgQt.pc
    /mnt/vcpkg-ci/packages/osg-qt_x64-linux/debug/lib64/pkgconfig/openscenegraph-osgQt.pc

You can move the pkgconfig files with the following commands:

    file(INSTALL "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig/name.pc" DESTINATION "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")
    file(REMOVE_RECURSE "${CURRENT_PACKAGES_DIR}/a/path/pkgconfig")

#21048

REGRESSION: qca:x64-linux. If expected, add qca:x64-linux=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-osx. If expected, add qca:x64-osx=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-windows-static-md. If expected, add qca:x64-windows-static-md=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-windows-static. If expected, add qca:x64-windows-static=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-windows. If expected, add qca:x64-windows=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x86-windows. If expected, add qca:x86-windows=fail to .\scripts\ci.baseline.txt.

CMake Error in plugins/qca-botan/CMakeLists.txt:
  Imported target "PkgConfig::BOTAN" includes non-existent path

    "/mnt/vcpkg-ci/installed/x64-linux/debug/lib/pkgconfig/../../../include/botan-2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Error in plugins/qca-botan/CMakeLists.txt:
  Imported target "PkgConfig::BOTAN" includes non-existent path

    "/mnt/vcpkg-ci/installed/x64-linux/debug/lib/pkgconfig/../../../include/botan-2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.



CMake Warning:
  Manually-specified variables were not used by the project:

    QT4_BUILD
    VCPKG_PLATFORM_TOOLSET
    VCPKG_SET_CHARSET_FLAG
    _VCPKG_ROOT_DIR

I'm pretty sure this is #20977

@BillyONeal BillyONeal added requires:author-response and removed depends:different-pr This PR or Issue depends on a PR which has been filed labels Oct 28, 2021
@BillyONeal BillyONeal added depends:different-pr This PR or Issue depends on a PR which has been filed and removed requires:author-response labels Oct 29, 2021
@BillyONeal
Copy link
Copy Markdown
Member Author

@BillyONeal BillyONeal removed the depends:different-pr This PR or Issue depends on a PR which has been filed label Oct 29, 2021
@BillyONeal
Copy link
Copy Markdown
Member Author

I'm pretty sure this is #20977

I checked that it works now.

@BillyONeal
Copy link
Copy Markdown
Member Author

PASSING, REMOVE FROM FAIL LIST: hpx:x64-linux (.\scripts\ci.baseline.txt)

Yay!

REGRESSION: log4cxx:x64-osx. If expected, add log4cxx:x64-osx=fail to .\scripts\ci.baseline.txt.

Building package log4cxx[core]:x64-osx...
-- Downloading https://archive.apache.org/dist/logging/log4cxx/0.12.0/apache-log4cxx-0.12.0.tar.gz -> apache-log4cxx-0.12.0.tar.gz...
-- Extracting source /Users/vagrant/Data/downloads/apache-log4cxx-0.12.0.tar.gz
-- Applying patch expat.patch
-- Applying patch linux.patch
-- Applying patch pkgconfig.patch
-- Using source at /Users/vagrant/Data/buildtrees/log4cxx/src/0.12.0-99c0951785.clean
-- Found external ninja('1.10.2').
-- Configuring x64-osx-dbg
-- Configuring x64-osx-rel
-- Building x64-osx-dbg
-- Building x64-osx-rel
-- Fixing pkgconfig file: /Users/vagrant/Data/packages/log4cxx_x64-osx/lib/pkgconfig/liblog4cxx.pc
CMake Error at scripts/cmake/vcpkg_fixup_pkgconfig.cmake:78 (message):
  /usr/local/bin/pkg-config --exists liblog4cxx failed with error code: 1

      ENV{PKG_CONFIG_PATH}: "/Users/vagrant/Data/packages/log4cxx_x64-osx/lib/pkgconfig:/Users/vagrant/Data/packages/log4cxx_x64-osx/share/pkgconfig:/Users/vagrant/Data/installed/x64-osx/lib/pkgconfig:/Users/vagrant/Data/installed/x64-osx/share/pkgconfig"
      output: Package 'liblog4cxx' has no Name: field
Call Stack (most recent call first):
  scripts/cmake/vcpkg_fixup_pkgconfig.cmake:201 (z_vcpkg_fixup_pkgconfig_check_files)
  ports/log4cxx/portfile.cmake:31 (vcpkg_fixup_pkgconfig)
  scripts/ports.cmake:142 (include)

Looks like this port is making bad pkgconfig files, but that's not a regression from this change.

PASSING, REMOVE FROM FAIL LIST: ompl:x64-linux (.\scripts\ci.baseline.txt)
PASSING, REMOVE FROM FAIL LIST: ompl:x64-osx (.\scripts\ci.baseline.txt)
REGRESSION: omplapp:x64-linux. If expected, add omplapp:x64-linux=fail to .\scripts\ci.baseline.txt.
REGRESSION: omplapp:x64-osx. If expected, add omplapp:x64-osx=fail to .\scripts\ci.baseline.txt.

ompl is fixed and now things that depend on it are broken, so not a regression from this change.

REGRESSION: qca:x64-linux. If expected, add qca:x64-linux=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-osx. If expected, add qca:x64-osx=fail to .\scripts\ci.baseline.txt.
REGRESSION: qca:x64-windows
REGRESSION: qca:x64-windows-static
REGRESSION: qca:x86-windows

  Imported target "PkgConfig::BOTAN" includes non-existent path

    "D:/installed/x86-windows/lib/pkgconfig/../../include/botan-2"

  in its INTERFACE_INCLUDE_DIRECTORIES.  Possible reasons include:

  * The path was deleted, renamed, or moved to another location.

  * An install or uninstall procedure did not complete successfully.

  * The installation package was faulty and references files it does not
  provide.

This looks like an incorrectly working optional dependency in qca that causes it to fail if botan is also installed, not a regression from this change.

REGRESSION: rtaudio:x64-linux. If expected, add rtaudio:x64-linux=fail to .\scripts\ci.baseline.txt.

[12/20] : && /usr/bin/c++ -fPIC -Wall -Werror -g  tests/CMakeFiles/playsaw.dir/playsaw.cpp.o -o tests/playsaw  librtaudio.a  /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a  -lpthread && :
FAILED: tests/playsaw 
: && /usr/bin/c++ -fPIC -Wall -Werror -g  tests/CMakeFiles/playsaw.dir/playsaw.cpp.o -o tests/playsaw  librtaudio.a  /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a  -lpthread && :
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(dlmisc.o): in function `snd_dlinfo_origin':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:72: undefined reference to `dladdr1'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:74: undefined reference to `dlinfo'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(dlmisc.o): in function `__snd_dlopen':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:155: undefined reference to `dlopen'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:161: undefined reference to `dlerror'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(dlmisc.o): in function `snd_dlclose':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:191: undefined reference to `dlclose'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(dlmisc.o): in function `snd_dlsym_verify':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:221: undefined reference to `dlsym'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(dlmisc.o): in function `snd_dlsym':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/.././../src/v1.2.5.1-98fc6d1525.clean/src/dlmisc.c:269: undefined reference to `dlsym'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(pcm_ladspa.o): in function `snd_pcm_ladspa_free_plugins':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:188: undefined reference to `dlclose'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(pcm_ladspa.o): in function `snd_pcm_ladspa_check_file':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:1094: undefined reference to `dlopen'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:1096: undefined reference to `dlsym'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:1114: undefined reference to `dlclose'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:1131: undefined reference to `dlclose'
/usr/bin/ld: /mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_ladspa.c:1139: undefined reference to `dlclose'
/usr/bin/ld: /mnt/vcpkg-ci/installed/x64-linux/debug/lib/libasound.a(pcm_meter.o): in function `snd_pcm_meter_add_scope_conf':
/mnt/vcpkg-ci/buildtrees/alsa/x64-linux-dbg/src/pcm/../.././../src/v1.2.5.1-98fc6d1525.clean/src/pcm/pcm_meter.c:676: undefined reference to `dlsym'
collect2: error: ld returned 1 exit status

Don't know exactly what this is but doesn't look related.

@BillyONeal BillyONeal merged commit 5bdb9d6 into microsoft:master Oct 30, 2021
@BillyONeal BillyONeal deleted the download-script branch October 30, 2021 05:39
@Neumann-A
Copy link
Copy Markdown
Contributor

The last one is a) missing explicit dependency on alsa and b) missing linkage to dl.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

category:infrastructure Pertaining to the CI/Testing infrastrucutre category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants