ci: Define cases when invalidate vcpkg binary cache #23329

hebasto · 2021-10-20T19:30:16Z

On master (c8bae2b), the size of the vcpkg binary cache is potentially unbounded.

The reason of such behavior is the internal caching logic, following which, any change in the set of parameters defined by the vcpkg implementation will cause compiling of a new binaries following by adding them to the current cache.

This PR defines two obvious cases when the vcpkg binary cache will be invalidated.

The VCPKG_TAG variable renamed to CI_VCPKG_TAG to prevent any possible name clash with vcpkg-specific variables. The vcpkg_cache script renamed into more meaningful one.

This change comes with two improvements: 1) using the VCPKG_DEFAULT_BINARY_CACHE variable drops dependency on vcpkg default cached archives location, and improves readability 2) two obvious cases when binary cache is invalidated are defined, that guaranties it won't grow boundlessly when, for example, vcpkg has being updated.

fanquake · 2021-10-20T23:22:31Z

Isn't this almost the same as #23215? Why can't it be done in that PR? It's nearly impossible to keep track of the constant changes for these Windows CIs.

hebasto · 2021-10-21T06:30:08Z

It's nearly impossible to keep track of the constant changes for these Windows CIs.

Sorry about that.

Isn't this almost the same as #23215?

Yes, this PR is split from #23215.

Why can't it be done in that PR?

That was my initial idea when I opened #23215. But it created a confusion (at least, I have such an impression), that the goal of such combined PR was introducing a vcpkg binary caching. It is not the case. So I decided to leave #23215 solely on purpose of introducing of vcpkg downloads cache which has completely different goal in comparison to binary cache.

Therefore, this PR is focused on an improvement of already used vcpkg binary caching feature.

And #23215 will be focused on an introducing a new vcpkg downloads cache.

hebasto · 2021-10-21T06:30:58Z

Friendly ping @sipsorcery 🐅

sipsorcery · 2021-10-21T17:52:38Z

@hebasto are you sure that you're caching the correct vcpkg directory?

It's my understanding that the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives will hold the zipped source files that will then need to be built.

On my local machine the vcpkg manifest mechanism puts the built binaries in build_msvc/vcpkg_installed.

hebasto · 2021-10-21T20:24:53Z

@sipsorcery

@hebasto are you sure that you're caching the correct vcpkg directory?

Yes, I am 🐅

One could track any dependency package (berkeleydb, for example):

16df28c, the binary cache is unpopulated, i.e., C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives is empty:

In the build log one can see:

...
  The following packages will be built and installed:
      berkeleydb[core]:x64-windows-static -> 4.8.30#5
...
  Could not locate cached archive: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
...
  Starting package 1/80: berkeleydb:x64-windows-static
  Building package berkeleydb[core]:x64-windows-static...
  -- Downloading http://download.oracle.com/berkeley-db/db-4.8.30.NC.zip -> db-4.8.30.NC.zip...
  -- Extracting source C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/downloads/db-4.8.30.NC.zip
  -- Applying patch fix-conflict-macro.patch
  -- Using source at C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/buildtrees/berkeleydb/src/db-4-87fc9a5730.clean
  -- Found external ninja('1.10.2').
  -- Configuring x64-windows-static
  -- Building x64-windows-static-rel
  -- Installing: C:/Users/ContainerAdministrator/AppData/Local/Temp/vcpkg/packages/berkeleydb_x64-windows-static/share/berkeleydb/copyright
  -- Performing post-build validation
  -- Performing post-build validation done
  Stored binary cache: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
  Building package berkeleydb[core]:x64-windows-static... done
  Installing package berkeleydb[core]:x64-windows-static...
  Installing package berkeleydb[core]:x64-windows-static... done
  Elapsed time for package berkeleydb:x64-windows-static: 52.01 s
...

a7f28af, now the binary cache is fully populated, and in the build log:

...
  The following packages will be built and installed:
      berkeleydb[core]:x64-windows-static -> 4.8.30#5
...
  Using cached binary package: C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip
...
  Starting package 1/80: berkeleydb:x64-windows-static
  Building package berkeleydb[core]:x64-windows-static...
  Building package berkeleydb[core]:x64-windows-static... done
  Installing package berkeleydb[core]:x64-windows-static...
  Installing package berkeleydb[core]:x64-windows-static... done
  Elapsed time for package berkeleydb:x64-windows-static: 27.04 ms
...

It's my understanding that the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives will hold the zipped source files that will then need to be built.

Actually, the C:\Users\ContainerAdministrator\AppData\Local\vcpkg\archives\62\62c79c2f0c0d9d79208917f62e9cc30952a57354.zip archive also contains a compiled lib\libdb48.lib.

On my local machine the vcpkg manifest mechanism puts the built binaries in build_msvc/vcpkg_installed.

Correct. But this action seems orthogonal to the vcpkg "Binary Caching" mechanism.

sipsorcery · 2021-10-21T20:52:30Z

@hebasto are you sure that you're caching the correct vcpkg directory?

Yes, I am 🐅

I stand corrected. I did a build, deleted the build_msvc/vcpkg_installed directory, built again and the binary dependencies were copied from C:\Users\aaron\AppData\Local\vcpkg\archives.

Am I correct that the CI snippet below deletes and re-creates the %VCPKG_DEFAULT_BINARY_CACHE% directory is the msbuild version changes? And that's the mechanism to stop it growing when the vcpkg dependencies need to get rebuilt?

  vcpkg_binary_cache:
    folder: '%VCPKG_DEFAULT_BINARY_CACHE%'
    reupload_on_changes: true
    fingerprint_script:
      - echo %CI_VCPKG_TAG%
      - msbuild -version
    populate_script:
      - mkdir %VCPKG_DEFAULT_BINARY_CACHE%

hebasto · 2021-10-21T21:16:02Z

@sipsorcery

Am I correct that the CI snippet below deletes and re-creates the %VCPKG_DEFAULT_BINARY_CACHE% directory is the msbuild version changes?

To be precise, an old cache is not loaded, rather "deleted". Cases for such cache invalidation are any of the following: CI_VCPKG_TAG is changed and/or msbuild -version is changed.

And that's the mechanism to stop it growing when the vcpkg dependencies need to get rebuilt?

Yes, it is. But only in two explicit cases: CI_VCPKG_TAG is changed and/or msbuild -version is changed.

FWIW, usual binary cache expected size is 83 MB, on master (8a083bc) it is 334 MB now.

Just in case: https://cirrus-ci.org/guide/writing-tasks/#cache-instruction

sipsorcery · 2021-10-21T21:56:41Z

ACK e8692cf.

f778845 ci: Add vcpkg tools cache (Hennadii Stepanov) Pull request description: On master (02feae5) vcpkg downloads some tools that are used internally: ``` ... A suitable version of cmake was not found (required v3.20.0). Downloading portable cmake v3.20.0... Downloading cmake... https://github.com/Kitware/CMake/releases/download/v3.20.2/cmake-3.20.2-windows-i386.zip -> C:\Users\ContainerAdministrator\AppData\Local\Temp\vcpkg\downloads\cmake-3.20.2-windows-i386.zip Extracting cmake... A suitable version of 7zip was not found (required v18.1.0). Downloading portable 7zip v18.1.0... Downloading 7zip... https://www.nuget.org/api/v2/package/7-Zip.CommandLine/18.1.0 -> C:\Users\ContainerAdministrator\AppData\Local\Temp\vcpkg\downloads\7-zip.commandline.18.1.0.nupkg Extracting 7zip... A suitable version of nuget was not found (required v5.5.0). Downloading portable nuget v5.5.0... Downloading nuget... https://dist.nuget.org/win-x86-commandline/v5.5.1/nuget.exe -> C:\Users\ContainerAdministrator\AppData\Local\Temp\vcpkg\downloads\22ea847d-nuget.exe Detecting compiler hash for triplet x64-windows-static... A suitable version of powershell-core was not found (required v7.1.0). Downloading portable powershell-core v7.1.0... Downloading powershell-core... https://github.com/PowerShell/PowerShell/releases/download/v7.1.3/PowerShell-7.1.3-win-x86.zip -> C:\Users\ContainerAdministrator\AppData\Local\Temp\vcpkg\downloads\PowerShell-7.1.3-win-x86.zip Extracting powershell-core... ... ``` If any of downloads above fails the whole CI task fails (see #23107). The most recent failure examples in the master branch: - c001da3, [log](https://api.cirrus-ci.com/v1/task/5182966176415744/logs/build.log) - 12ff899, [log](https://api.cirrus-ci.com/v1/task/5367684129882112/logs/build.log) This PR adds vcpkg tools cache. Closes #23107. --- Note for reviewers. Some patches from the initial PR were split into separated PRs: #23310 and #23329. Therefore, a discussion here could be outdated or irrelevant until the recent [push](#23215 (comment)). ACKs for top commit: sipsorcery: ACK f778845. Tree-SHA512: 201f4e4d38c00cbec6aaeca4f3e1b60f1c65289fb68b82cead47f37f6af5ac42dd539cf8ed3566f5bd9afe4a7762d42bbabb316d58f47352d4e7bfc2214806fe

hebasto added 2 commits October 20, 2021 22:16

ci, refactor: Rename VCPKG_TAG variable and vcpkg_cache script

b00646b

The VCPKG_TAG variable renamed to CI_VCPKG_TAG to prevent any possible name clash with vcpkg-specific variables. The vcpkg_cache script renamed into more meaningful one.

hebasto added Tests Windows labels Oct 20, 2021

fanquake merged commit 81cdbec into bitcoin:master Oct 22, 2021

hebasto deleted the 211020-vcpkg branch October 22, 2021 08:52

hebasto mentioned this pull request Oct 22, 2021

ci: Add vcpkg tools cache #23215

Merged

bitcoin locked and limited conversation to collaborators Oct 30, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

ci: Define cases when invalidate vcpkg binary cache #23329

ci: Define cases when invalidate vcpkg binary cache #23329

Uh oh!

hebasto commented Oct 20, 2021

Uh oh!

fanquake commented Oct 20, 2021

Uh oh!

hebasto commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021 •

edited

Loading

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021 •

edited

Loading

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ci: Define cases when invalidate vcpkg binary cache #23329

ci: Define cases when invalidate vcpkg binary cache #23329

Uh oh!

Conversation

hebasto commented Oct 20, 2021

Uh oh!

fanquake commented Oct 20, 2021

Uh oh!

hebasto commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

hebasto commented Oct 21, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sipsorcery commented Oct 21, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

hebasto commented Oct 21, 2021 •

edited

Loading

hebasto commented Oct 21, 2021 •

edited

Loading