Skip to content

GHA: fix caching old mingw-w64 toolchains in the Windows workflow#13856

Closed
vszakats wants to merge 3 commits intocurl:masterfrom
vszakats:gha-cache-test
Closed

GHA: fix caching old mingw-w64 toolchains in the Windows workflow#13856
vszakats wants to merge 3 commits intocurl:masterfrom
vszakats:gha-cache-test

Conversation

@vszakats
Copy link
Member

@vszakats vszakats commented Jun 2, 2024

  • stop altering the PATH via GITHUB_ENV. This confused the
    actions/cache post-job, which needs to run in the exact same
    environment as its pre-job, to have a consistent cache entry "version"
    hash. Altering the PATH via GITHUB_ENV spills into the the
    post-job and breaks this hash. GHA doesn't reset the env automatically
    and I have not found a way to do it manually.

  • add double-quotes where missing.

  • move cache directory under USERPROFILE to not rely on absolute
    paths.

  • make cache directory flatter and versionless.

Follow-up to 0914d8a #13759
Closes #13856

@vszakats vszakats added Windows Windows-specific CI Continuous Integration labels Jun 2, 2024
@vszakats vszakats marked this pull request as draft June 2, 2024 08:30
@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

Meanwhile caches are almost never hit though the key exists, and often still attempting to save the cache (and often failing), GitHub accumulated a large collection of these Windows caches. The behaviour is explained here, though it assumes that a key cached on the main branch is found by branches, which they don't seem to be for the curl Windows workflow:
actions/cache#968

There is a list of Windows-related issues in the action's repo, this one is still open:
actions/cache#1275

Though in curl's case a cache write also fails often and inexplicably.

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

I'm manually purging a bunch of them to go back under 10GB total.

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

cache@v2 seemed to save the cache then find the key correctly, but then blew up when extracting the cache:

Received 0 of 171720426 (0.0%), 0.0 MBs/sec
Received 96468992 of 171720426 (56.2%), 45.8 MBs/sec
Received 171720426 of 171720426 (100.0%), 54.4 MBs/sec
Cache Size: ~164 MB (171720426 B)
C:\Windows\System32\tar.exe -z -xf D:/a/_temp/54d39e44-ee22-4320-b624-a40209da2be5/cache.tgz -P -C D:/a/curl/curl
C:/my-cache/mingw-w64-9.5.0-x86_64/: Can't create '\\\\?\\C:\\my-cache\\mingw-w64-9.5.0-x86_64'
C:/my-cache/mingw-w64-9.5.0-x86_64/mingw64/: Can't create '\\\\?\\C:\\my-cache\\mingw-w64-9.5.0-x86_64\\mingw64'
C:/my-cache/mingw-w64-9.5.0-x86_64/mingw64/bin/: Can't create '\\\\?\\C:\\my-cache\\mingw-w64-9.5.0-x86_64\\mingw64\\bin'
C:/my-cache/mingw-w64-9.5.0-x86_64/mingw64/include/: Can't create '\\\\?\\C:\\my-cache\\mingw-w64-9.5.0-x86_64\\mingw64\\include'
[...]

Ref: https://github.com/curl/curl/actions/runs/9337833329/job/25700218626

This was the tar command:

C:\Windows\System32\tar.exe --posix -z -cf cache.tgz -P -C D:/a/curl/curl --files-from manifest.txt

Then back to square one:

Unable to reserve cache with key Windows-mingw-w64-6.4.0-i686, another job may be creating this cache.

Ref: https://github.com/curl/curl/actions/runs/9337833329/job/25700218908#step:20:2

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

v2 with unixy path notation "works" because there is nothing in the cache tarballs.

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

It seems that the cache "version" is different each time, which makes the action create new cache entries on every run:

##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/LBtO2h2PnCINU7gm3jjmEjodUj2g1Be2AKcmGQDkydACxmcFgb/_apis/artifactcache/cache?keys=Windows-mingw-w64-9.5.0-x86_64&version=07e247f9ca987b2b8bd2de1c6c67497de473063bc6bbfb69c0b3a8f85e4a763e
##[debug]Resource Url: https://acghubeus2.actions.githubusercontent.com/LBtO2h2PnCINU7gm3jjmEjodUj2g1Be2AKcmGQDkydACxmcFgb/_apis/artifactcache/caches?key=Windows-mingw-w64-9.5.0-x86_64
##[debug]No matching cache found for cache key 'Windows-mingw-w64-9.5.0-x86_64', version '07e247f9ca987b2b8bd2de1c6c67497de473063bc6bbfb69c0b3a8f85e4a763e and scope refs/pull/13856/merge. There exist one or more cache(s) with similar key but they have different version or scope. See more info on cache matching here: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows#matching-a-cache-key 
##[debug]Other caches with similar key:
##[debug]Cache Key: Windows-mingw-w64-9.5.0-x86_64, Cache Version: 8451cebe2ece5a08d76f850c57ae0e73d5576ede09333ab3e33d6c7842491cd2, Cache Scope: refs/pull/13856/merge, Cache Created: 2024-06-02T12:33:41.5333333Z
##[debug]Failed to delete archive: Error: ENOENT: no such file or directory, unlink ''
Cache not found for input keys: Windows-mingw-w64-9.5.0-x86_64

Ref: https://github.com/curl/curl/actions/runs/9338157000/job/25701293515#step:2:56

curl -L -A curl \
  -H "Accept: application/vnd.github+json" \
  -H "X-GitHub-Api-Version: 2022-11-28" \
  https://api.github.com/repos/curl/curl/actions/caches | jq

Manual:
https://github.com/actions/cache?tab=readme-ov-file#cache-version

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

It looks like this line in one of the step:

echo "PATH=$(cygpath "${USERPROFILE}")/my-cache/mingw64/bin:/c/msys64/usr/bin:$PATH" >> "$GITHUB_ENV"

makes the caching post-script confused, when it fails to find zstd, and which makes the "version" hash different from the pre-script, where zstd is present:

Post job cleanup.
##[debug]Checking zstd --quiet --version
##[debug]Unable to locate executable file: zstd. Please verify either the file path exists or the file can be found within a directory specified by the PATH environment variable. Also verify the file has a valid extension for an executable file.

Ref: https://github.com/curl/curl/actions/runs/9338773353/job/25702416548#step:16:30

while in the pre-script:

##[debug]Checking zstd --quiet --version
##[debug]1.5.6
##[debug]zstd version: 1.5.6

Ref: https://github.com/curl/curl/actions/runs/9339474086/job/25704019595#step:2:39

(This doesn't explain why it was working sometimes.)

I couldn't figure out how to undo the above line manually, rm -f "$GITHUB_ENV" did not have this effect.

After replacing the $GITHUB_ENV solution with just setting PATH in each step, the cache entry version number became consistently this value: b52f65273fbb3e28d605d17760124057cf29eed0a598bb5e72a3579ad685ec9a

Another, earlier issue was that ~ is interpreted as USERPROFILE in the path: directive, while under MSYS2 shell, it means HOME, which is a user home specific to the MSYS2 env. With that fixed, ~ is safe to use again.

@vszakats
Copy link
Member Author

vszakats commented Jun 2, 2024

Initial post for future reference:

Caching continues to fail on Windows workflows:

Cache not found for input keys: Windows-mingw-w64-9.5.0-x86_64
[...]
Failed to save: Unable to reserve cache with key Windows-mingw-w64-9.5.0-x86_64, another job may be creating this cache. More details: Cache already exists. Scope: refs/heads/master, Key: Windows-mingw-w64-9.5.0-x86_64, Version: 047366457c7c36fd25375c0f03064d9f4d03a10402d8339f2ac76a6fe85ea742

Ref: https://github.com/curl/curl/actions/runs/9334517760/job/25697464520?pr=13622#step:2:10

In the same PR, caching works fine on Linux:

Cache restored successfully
Cache restored from key: Linux-http3-build-cache-quictls-no-deprecated-3.1.4+quic
[...]
Cache hit occurred on the primary key Linux-http3-build-cache-quictls-no-deprecated-3.1.4+quic, not saving cache.

https://github.com/curl/curl/actions/runs/9334517768/job/25692846508?pr=13622#step:2:25

It's probably a stupid error on my part when implementing this.

Caching (almost?) always fails to find the cache:
https://github.com/curl/curl/actions/workflows/windows.yml

@vszakats vszakats changed the title test cache failing in Windows workflows GHA: fix caching old-mingw-w64 tools in the Windows workflow Jun 2, 2024
vszakats added 3 commits June 2, 2024 18:34
This makes it not rely on specific absolute paths.
GITHUB_ENV interferes with actions/cache's post-job, resulting in an unstable
cache entry "version" due to not finding the same tools (e.g. zstd) as in the
pre-job.
@vszakats vszakats changed the title GHA: fix caching old-mingw-w64 tools in the Windows workflow GHA: fix caching old-mingw-w64 toolchains in the Windows workflow Jun 2, 2024
@vszakats vszakats marked this pull request as ready for review June 2, 2024 16:51
@vszakats vszakats changed the title GHA: fix caching old-mingw-w64 toolchains in the Windows workflow GHA: fix caching old mingw-w64 toolchains in the Windows workflow Jun 2, 2024
@vszakats vszakats closed this in 1d63e33 Jun 2, 2024
@vszakats vszakats deleted the gha-cache-test branch June 2, 2024 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CI Continuous Integration Windows Windows-specific

Development

Successfully merging this pull request may close these issues.

1 participant