From #923:

As there is no visible ci going on I was wondering if the MS Team actually checks if all the portfiles work correctly (on which platform / with which compiler / are you running any tests / any test projects?). And if the library versions in the master branch play "nice" together or if there are any incompatibilities.
Especially when less experienced people like me are trying to write a non-trivial portfile or upgrading an existing one it would be nice to have more feedback than just: "It seems to work on my machine"

CI / unit testing / integration testing

While nowhere near as visible as it needs to be, we do have a limited amount of CI that runs on every change to the master branch. You can see some checkmarks beside many of the commits in https://github.com/Microsoft/vcpkg/commits/master. I've also just added the current status badge to our front-page README as well.

This CI system does the following (Win10+VS2017):

bootstrap
recursively purges all out-of-date libraries
build a "core" set, including their dependencies under several flavors
- This core set includes azure-storage-cpp opencv protobuf chakracore libuv and several others

[todo: we should check in the script for this, at least as documentation.]

We also do (irregularly) build all ports -- the results from last time showed 6 failing ports (and 3 cascaded) out of ~170 under x86-windows[1]. Most issues we see day-to-day are unfortunately ones that this style of CI system can't reasonably catch: they're due to unique combinations of client and port, such as having files under "C:\OpenSSL" (which confused find_package(OpenSSL) -- fixed!) or having a "site-config.jam" (which confused the boost build -- not fixed ☹️).

It would be really awesome to have a dashboard for this! Something like a "sticky" GitHub issue would be ideal, but unfortunately the closest we have to that is perhaps a wiki page? We need to do some research to find the best option. I'd really appreciate links to all your favorite projects in this regard -- who does this the best?

Testing techniques

Today we don't run any tests beyond the build+post-package lint for libraries built inside Vcpkg. This is definitely an area which needs improvement! There are a few obvious ways we can improve testing:

Use the tests inside the library -- what to do about flaky tests?
Provide an example program for each port -- would serve as great documentation!
Build all the ports that depend on the "port under test" -- the CI system above does this for core libraries like boost and openssl, but it would be good to expand this for all update PRs
Expand our automatic post-package tests -- these are great for catching packaging errors like "I accidentally built the wrong architecture" or "I forgot to copy over the include files"

As mentioned in a previous comment, there are some additional core capabilities we need to support some of these. We would need to expand the buildsystem integration to allow pulling from a package's temporary directory before installation.

[1] The failing ports were:
clockutils (since fixed)
hdf5 (since fixed)
libmysql (since fixed)
nana (since fixed)
qt5 (since fixed)
spirv-tools

Find a testing solution #72

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions